2024-03-25T16:14:46.608 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:14:46.609 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/orig.config.yaml 2024-03-25T16:14:46.651 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:14:46.738 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621260' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDyGPtVmN/t/hScr/Lh41lhd2ESJ/noBK6bWwPWjF49cnIjlANMDM1i0OWQJ541/5QR58IysledB17AshalkgKs= smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5lvloWX4M/z9MMgX4Co9KLPl1cnAOYTbRK0YDa1V8wVIVlmbo+CfY4RPrgGwOTj9EaNdDBbQRWlsZTtY4Fd28= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:14:46.739 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:14:46.739 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:14:46.739 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:14:46.741 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:14:46.764 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:14:46.765 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:14:46.765 INFO:teuthology.packaging:ref: None 2024-03-25T16:14:46.765 INFO:teuthology.packaging:tag: None 2024-03-25T16:14:46.765 INFO:teuthology.packaging:branch: main 2024-03-25T16:14:46.765 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:14:46.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:14:47.028 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:14:47.029 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:14:47.035 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:14:47.036 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:14:47.043 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:14:47.059 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:14:47.066 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:14:47.091 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:38.570206', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDyGPtVmN/t/hScr/Lh41lhd2ESJ/noBK6bWwPWjF49cnIjlANMDM1i0OWQJ541/5QR58IysledB17AshalkgKs='} 2024-03-25T16:14:47.116 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:38.568797', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5lvloWX4M/z9MMgX4Co9KLPl1cnAOYTbRK0YDa1V8wVIVlmbo+CfY4RPrgGwOTj9EaNdDBbQRWlsZTtY4Fd28='} 2024-03-25T16:14:47.116 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:14:47.122 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:14:47.123 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:14:47.123 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:14:47.197 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2aa99800d0>, signals=[15]) 2024-03-25T16:14:47.198 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:14:47.204 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:14:47.204 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T16:14:47.208 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:47.298 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T16:14:47.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:47.389 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:14:47.395 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2024-03-25T16:14:47.414 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2024-03-25T16:14:47.414 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="9" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="9" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:14:47.472 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2024-03-25T16:14:47.473 INFO:teuthology.orchestra.run.smithi132.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:14:47.473 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:14:47.473 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:14:47.473 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:14:47.473 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:14:47.473 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:14:47.473 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2024-03-25T16:14:47.497 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-03-25T16:14:47.515 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-03-25T16:14:47.515 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-03-25T16:14:47.571 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:14:47.572 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:14:47.573 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:14:47.573 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-03-25T16:14:47.595 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:14:47.604 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:14:47.611 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:14:47.611 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:14:47.614 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:14:47.629 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:14:47.637 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:14:47.638 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:14:47.670 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:14:47.955 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:14:48.014 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:14:48.029 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:14:48.029 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:14:48.029 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:14:48.030 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:14:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:48.049 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-03-25T16:14:48.049 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-25T16:14:48.107 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:14:48.107 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.14.0-361.el9.x86_64 2024-03-25T16:14:48.107 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2024-03-25T16:14:48.108 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:14:48.108 INFO:teuthology.task.kernel:Running kernel on smithi132: 5.14.0-361.el9.x86_64 2024-03-25T16:14:48.108 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2024-03-25T16:14:50.441 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-25T16:14:50.688 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 7.3 MB/s | 6.7 MB 00:00 2024-03-25T16:14:51.816 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-03-25T16:14:52.296 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-25T16:14:52.573 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-25T16:14:54.877 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-25T16:15:02.077 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 875 kB/s | 8.0 MB 00:09 2024-03-25T16:15:06.891 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 756 kB/s | 6.1 MB 00:08 2024-03-25T16:15:08.333 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-25T16:15:11.079 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 7.9 MB/s | 21 MB 00:02 2024-03-25T16:15:12.127 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:08 2024-03-25T16:15:16.076 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:15:17.047 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-25T16:15:18.224 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:15:18.225 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:15:18.312 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:15:18.317 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout:Install 4 Packages 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 76 M 2024-03-25T16:15:18.318 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 123 M 2024-03-25T16:15:18.319 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:15:18.544 INFO:teuthology.orchestra.run.smithi161.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 571 kB/s | 44 kB 00:00 2024-03-25T16:15:19.202 INFO:teuthology.orchestra.run.smithi161.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-25T16:15:19.420 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-25T16:15:19.777 INFO:teuthology.orchestra.run.smithi161.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-25T16:15:20.488 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T16:15:21.278 INFO:teuthology.orchestra.run.smithi161.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 10 MB/s | 27 MB 00:02 2024-03-25T16:15:21.278 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:21.278 INFO:teuthology.orchestra.run.smithi161.stdout:Total 26 MB/s | 76 MB 00:02 2024-03-25T16:15:21.940 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:15:22.304 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:15:22.304 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:15:23.294 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:15:23.294 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:15:25.447 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-03-25T16:15:25.810 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:15:26.476 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:26.561 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:27.479 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:15:27.479 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:15:27.567 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:15:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:15:27.575 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:15:27.575 INFO:teuthology.orchestra.run.smithi132.stdout:Install 4 Packages 2024-03-25T16:15:27.575 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:15:27.575 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 76 M 2024-03-25T16:15:27.575 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 123 M 2024-03-25T16:15:27.575 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:15:27.873 INFO:teuthology.orchestra.run.smithi132.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 425 kB/s | 44 kB 00:00 2024-03-25T16:15:28.657 INFO:teuthology.orchestra.run.smithi132.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-25T16:15:29.124 INFO:teuthology.orchestra.run.smithi132.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 25 MB/s | 33 MB 00:01 2024-03-25T16:15:29.329 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:29.473 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:30.257 INFO:teuthology.orchestra.run.smithi132.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T16:15:30.258 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:30.258 INFO:teuthology.orchestra.run.smithi132.stdout:Total 28 MB/s | 76 MB 00:02 2024-03-25T16:15:30.925 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:15:31.289 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:15:31.289 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:15:32.314 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:15:32.314 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:15:34.210 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:34.324 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:34.788 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:15:35.386 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:35.483 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:38.180 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:38.338 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:39.080 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:43.053 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:43.089 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:47.833 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:08.384 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:08.384 INFO:teuthology.orchestra.run.smithi161.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:16:08.385 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:16:08.429 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:08.756 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:09.123 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:09.123 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:16:09.123 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:16:09.123 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:09.123 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:09.123 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:16:09.124 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:09.124 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:16:09.124 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:16:09.459 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:09.564 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:09.564 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:09.564 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:16:09.564 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:16:09.564 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:16:09.564 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:16:09.565 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:16:09.565 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:09.645 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:09.645 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:09.645 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2024-03-25T16:16:09.690 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2024-03-25T16:16:09.690 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2024-03-25T16:16:09.690 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.06 2024-03-25T16:16:09.690 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 68.el9 2024-03-25T16:16:09.690 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2024-03-25T16:16:09.690 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Group : Unspecified 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 8264254 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : builder@centos.org 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2024-03-25T16:16:09.691 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:16:09.692 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:16:09.692 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:16:09.692 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:16:09.692 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2024-03-25T16:16:09.692 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:16:09.692 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:16:09.693 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:16:09.693 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:16:09.693 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:16:10.258 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2024-03-25T16:16:11.628 INFO:teuthology.orchestra.run.smithi161.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:16:11.665 INFO:teuthology.orchestra.run.smithi161.stderr:done 2024-03-25T16:16:11.668 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-03-25T16:16:11.683 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.DJXzemVbHX 2024-03-25T16:16:11.683 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DJXzemVbHX 2024-03-25T16:16:11.751 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.DJXzemVbHX 2024-03-25T16:16:11.892 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.DJXzemVbHX is 6KB 2024-03-25T16:16:11.942 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.DJXzemVbHX 2024-03-25T16:16:11.956 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:16:12.021 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:16:12.021 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:16:12.021 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:16:12.021 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:16:12.022 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:16:12.162 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2024-03-25T16:16:17.158 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:17.158 INFO:teuthology.orchestra.run.smithi132.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:16:17.158 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:16:17.203 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:17.607 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:17.607 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:17.607 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:17.608 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:16:18.031 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:16:18.347 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:18.447 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.448 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.448 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.448 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:16:18.448 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:16:18.448 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:16:18.448 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:16:18.449 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:18.528 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.529 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.529 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2024-03-25T16:16:18.567 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2024-03-25T16:16:18.567 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2024-03-25T16:16:18.567 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.06 2024-03-25T16:16:18.567 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 68.el9 2024-03-25T16:16:18.567 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2024-03-25T16:16:18.567 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Group : Unspecified 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 8264254 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : builder@centos.org 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:16:18.568 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:16:18.569 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:16:18.569 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:16:18.569 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2024-03-25T16:16:18.569 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:16:18.569 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:16:18.570 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:16:18.570 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.570 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:16:19.186 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2024-03-25T16:16:20.610 INFO:teuthology.orchestra.run.smithi132.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:16:20.647 INFO:teuthology.orchestra.run.smithi132.stderr:done 2024-03-25T16:16:20.649 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-25T16:16:20.664 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.GMQbfCkDSz 2024-03-25T16:16:20.665 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GMQbfCkDSz 2024-03-25T16:16:20.742 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.GMQbfCkDSz 2024-03-25T16:16:20.886 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.GMQbfCkDSz is 6KB 2024-03-25T16:16:20.939 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.GMQbfCkDSz 2024-03-25T16:16:20.953 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:16:21.018 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:16:21.018 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:16:21.018 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:16:21.018 INFO:teuthology.orchestra.run.smithi132.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:16:21.019 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:16:21.172 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2024-03-25T16:16:42.186 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:16:42.187 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:16:42.188 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:51.182 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:16:51.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-25T16:16:51.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:00.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-25T16:17:09.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-03-25T16:17:09.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:17:09.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:12.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-25T16:17:18.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-25T16:17:18.667 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:21.730 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2024-03-25T16:17:24.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:17:24.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:33.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-25T16:17:33.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:43.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-25T16:17:58.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:17:58.211 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:58.726 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-25T16:17:59.148 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:17:59.149 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:17:59.149 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-03-25T16:17:59.164 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:17:59.164 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:17:59.164 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:17:59.165 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:18:00.165 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:18:00.165 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:18:00.226 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-03-25T16:18:00.253 DEBUG:teuthology.parallel:result is None 2024-03-25T16:18:05.566 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-25T16:18:06.203 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-25T16:18:06.203 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:18:06.203 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2024-03-25T16:18:06.231 INFO:teuthology.orchestra.run.smithi132.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:18:06.231 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:18:06.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:18:06.231 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:18:07.232 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:18:07.232 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:18:07.545 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2024-03-25T16:18:07.603 DEBUG:teuthology.parallel:result is None 2024-03-25T16:18:07.603 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:18:07.610 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:18:07.611 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:18:07.613 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:18:07.632 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:18:07.639 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:18:07.646 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:18:07.646 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:18:07.668 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:18:07.791 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:18:07.799 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:18:07.799 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:18:07.801 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:18:07.835 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.845 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.849 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.857 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.859 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:18:07.867 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:18:07.867 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:18:07.869 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:18:07.922 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:18:07.932 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:18:07.932 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:18:07.936 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:18:07.979 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:08.031 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:08.083 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:08.167 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:08.207 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T16:18:08.207 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:18:08.274 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:08.315 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:08.375 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:08.461 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:08.504 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T16:18:08.504 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:18:08.572 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2024-03-25T16:18:08.574 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-03-25T16:18:08.615 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:18:08.670 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:18:09.041 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:18:09.048 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:18:09.049 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:18:09.059 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:18:09.068 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2024-03-25T16:18:09.143 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2024-03-25T16:18:09.145 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-03-25T16:18:09.223 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-03-25T16:18:09.225 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:18:09.225 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2024-03-25T16:18:09.247 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2024-03-25T16:18:09.248 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-03-25T16:18:09.276 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-03-25T16:18:09.277 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi161.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:18:09.277 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:18:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:18:09.315 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:18:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:18:09.345 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:18:09.346 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:18:09.391 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:18:09.427 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:18:09.437 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:18:09.457 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:18:09.459 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi132.front.sepia.ceph.com,smithi161.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:22:25.424 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi161.front.sepia.ceph.com')] 2024-03-25T16:22:25.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-25T16:22:25.427 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:25.517 DEBUG:teuthology.orchestra.run.smithi132:> true 2024-03-25T16:22:25.596 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2024-03-25T16:22:25.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:22:25.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:25.685 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-25T16:22:25.765 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-25T16:22:25.765 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:22:25.786 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:22:25.786 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:22:25.787 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:22:25.789 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:22:25.789 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:22:25.818 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:22:25.837 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:22:25.838 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:22:25.852 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:22:25.863 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2024-03-25T16:22:25.877 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2024-03-25T16:22:25.881 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-03-25T16:22:25.893 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:22:25.895 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-03-25T16:22:25.909 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:22:25.912 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:22:25.928 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:22:25.961 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:22:25.965 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:22:25.965 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-25T16:22:25.965 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.965 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.965 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.966 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:22:25.993 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:22:25.993 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-25T16:22:25.993 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.993 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.993 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.994 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:25.994 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:22:26.002 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:22:26.002 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:22:26.002 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:22:26.003 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:22:26.008 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:22:26.008 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:22:26.009 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:22:26.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:22:26.011 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:22:26.012 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:22:26.152 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:22:26.152 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:22:26.221 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:22:26.221 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:22:26.554 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:22:26.554 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T16:22:26.554 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:22:26.584 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:22:26.584 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:22:26.617 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:22:26.617 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T16:22:26.617 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:22:26.645 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:22:26.645 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:22:26.653 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:22:26.713 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:22:26.735 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:22:26.792 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:22:26.802 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-03-25T16:22:26.803 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-03-25T16:22:26.858 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2024-03-25T16:22:26.860 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2024-03-25T16:22:27.041 INFO:teuthology.orchestra.run.smithi161.stdout:74 files removed 2024-03-25T16:22:27.069 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw 2024-03-25T16:22:27.106 INFO:teuthology.orchestra.run.smithi132.stdout:74 files removed 2024-03-25T16:22:27.133 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2024-03-25T16:22:28.387 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T16:22:28.393 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 80 kB/s | 82 kB 00:01 2024-03-25T16:22:29.237 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:22:29.252 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T16:22:30.005 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 2.6 kB/s | 1.9 kB 00:00 2024-03-25T16:22:30.053 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T16:22:30.424 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:22:30.513 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T16:22:31.955 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-25T16:22:32.091 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-25T16:22:35.673 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.0 MB 00:03 2024-03-25T16:22:39.724 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-03-25T16:22:41.303 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 911 kB/s | 8.0 MB 00:09 2024-03-25T16:22:43.766 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-25T16:22:46.234 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T16:22:48.088 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-25T16:22:53.052 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T16:22:55.480 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:55.495 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:22:55.496 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:22:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:22:55.498 INFO:teuthology.orchestra.run.smithi132.stdout:Install 30 Packages 2024-03-25T16:22:55.499 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2024-03-25T16:22:55.499 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:22:55.499 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 58 M 2024-03-25T16:22:55.499 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:23:04.126 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 885 kB/s | 19 MB 00:21 2024-03-25T16:23:08.421 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 7.2 MB/s | 6.1 MB 00:00 2024-03-25T16:23:09.518 INFO:teuthology.orchestra.run.smithi132.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 400 kB/s | 5.3 MB 00:13 2024-03-25T16:23:09.635 INFO:teuthology.orchestra.run.smithi132.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T16:23:10.203 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-03-25T16:23:11.137 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-25T16:23:11.760 INFO:teuthology.orchestra.run.smithi132.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 349 kB/s | 742 kB 00:02 2024-03-25T16:23:13.318 INFO:teuthology.orchestra.run.smithi132.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 571 kB/s | 9.7 MB 00:17 2024-03-25T16:23:13.344 INFO:teuthology.orchestra.run.smithi132.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 315 kB/s | 498 kB 00:01 2024-03-25T16:23:13.544 INFO:teuthology.orchestra.run.smithi132.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 226 kB/s | 45 kB 00:00 2024-03-25T16:23:13.936 INFO:teuthology.orchestra.run.smithi132.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 335 kB/s | 131 kB 00:00 2024-03-25T16:23:14.428 INFO:teuthology.orchestra.run.smithi132.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 329 kB/s | 162 kB 00:00 2024-03-25T16:23:15.145 INFO:teuthology.orchestra.run.smithi132.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 447 kB/s | 320 kB 00:00 2024-03-25T16:23:15.778 INFO:teuthology.orchestra.run.smithi132.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 470 kB/s | 298 kB 00:00 2024-03-25T16:23:15.978 INFO:teuthology.orchestra.run.smithi132.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 497 kB/s | 99 kB 00:00 2024-03-25T16:23:16.087 INFO:teuthology.orchestra.run.smithi132.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T16:23:16.120 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:23:16.137 INFO:teuthology.orchestra.run.smithi132.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:23:16.171 INFO:teuthology.orchestra.run.smithi132.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:23:16.230 INFO:teuthology.orchestra.run.smithi132.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:23:16.272 INFO:teuthology.orchestra.run.smithi132.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:23:16.305 INFO:teuthology.orchestra.run.smithi132.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:23:16.622 INFO:teuthology.orchestra.run.smithi132.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 683 kB/s | 216 kB 00:00 2024-03-25T16:23:16.723 INFO:teuthology.orchestra.run.smithi132.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 349 kB/s | 35 kB 00:00 2024-03-25T16:23:16.781 INFO:teuthology.orchestra.run.smithi132.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 766 kB/s | 44 kB 00:00 2024-03-25T16:23:16.957 INFO:teuthology.orchestra.run.smithi132.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-03-25T16:23:17.157 INFO:teuthology.orchestra.run.smithi132.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-03-25T16:23:17.216 INFO:teuthology.orchestra.run.smithi132.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-03-25T16:23:17.266 INFO:teuthology.orchestra.run.smithi132.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 976 kB/s | 49 kB 00:00 2024-03-25T16:23:17.308 INFO:teuthology.orchestra.run.smithi132.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-25T16:23:17.383 INFO:teuthology.orchestra.run.smithi132.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-03-25T16:23:17.450 INFO:teuthology.orchestra.run.smithi132.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:23:17.534 INFO:teuthology.orchestra.run.smithi132.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-25T16:23:18.583 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:23:18.600 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:23:18.601 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:23:18.602 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout:Install 30 Packages 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 2 Packages 2024-03-25T16:23:18.603 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:23:18.604 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 58 M 2024-03-25T16:23:18.604 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:23:21.825 INFO:teuthology.orchestra.run.smithi132.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 784 kB/s | 20 MB 00:25 2024-03-25T16:23:23.325 INFO:teuthology.orchestra.run.smithi132.stdout:(30/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 522 kB/s | 5.1 MB 00:10 2024-03-25T16:23:25.342 INFO:teuthology.orchestra.run.smithi132.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 911 kB/s | 3.1 MB 00:03 2024-03-25T16:23:27.967 INFO:teuthology.orchestra.run.smithi132.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 331 kB/s | 3.4 MB 00:10 2024-03-25T16:23:27.971 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:27.971 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.8 MB/s | 58 MB 00:32 2024-03-25T16:23:28.005 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:23:28.042 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:23:28.080 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:23:28.501 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:23:28.501 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:23:29.956 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:23:30.039 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:23:30.072 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:23:30.097 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:23:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:30.136 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:30.334 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:30.393 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:30.504 INFO:teuthology.orchestra.run.smithi161.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 467 kB/s | 5.3 MB 00:11 2024-03-25T16:23:30.629 INFO:teuthology.orchestra.run.smithi161.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 202 kB/s | 25 kB 00:00 2024-03-25T16:23:30.708 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:23:30.783 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:30.862 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:30.952 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:23:30.984 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:31.046 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:31.075 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:23:31.118 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:23:31.186 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:31.211 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:31.274 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:23:31.409 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:23:31.453 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:23:31.501 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:23:31.534 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:23:31.562 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:23:31.595 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:23:31.754 INFO:teuthology.orchestra.run.smithi161.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 659 kB/s | 742 kB 00:01 2024-03-25T16:23:31.901 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:23:31.925 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:31.946 INFO:teuthology.orchestra.run.smithi161.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 758 kB/s | 9.7 MB 00:13 2024-03-25T16:23:32.015 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:32.047 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:23:32.093 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:23:32.463 INFO:teuthology.orchestra.run.smithi161.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 704 kB/s | 498 kB 00:00 2024-03-25T16:23:32.542 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:23:32.571 INFO:teuthology.orchestra.run.smithi161.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 416 kB/s | 45 kB 00:00 2024-03-25T16:23:32.578 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:32.674 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:32.742 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:23:32.780 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:23:32.788 INFO:teuthology.orchestra.run.smithi161.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 606 kB/s | 131 kB 00:00 2024-03-25T16:23:33.066 INFO:teuthology.orchestra.run.smithi161.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 589 kB/s | 162 kB 00:00 2024-03-25T16:23:33.564 INFO:teuthology.orchestra.run.smithi161.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 641 kB/s | 320 kB 00:00 2024-03-25T16:23:33.956 INFO:teuthology.orchestra.run.smithi161.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 760 kB/s | 298 kB 00:00 2024-03-25T16:23:34.164 INFO:teuthology.orchestra.run.smithi161.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 477 kB/s | 99 kB 00:00 2024-03-25T16:23:34.273 INFO:teuthology.orchestra.run.smithi161.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T16:23:34.323 INFO:teuthology.orchestra.run.smithi161.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:23:34.357 INFO:teuthology.orchestra.run.smithi161.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:23:34.415 INFO:teuthology.orchestra.run.smithi161.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:23:34.457 INFO:teuthology.orchestra.run.smithi161.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:23:34.491 INFO:teuthology.orchestra.run.smithi161.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:23:34.840 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:34.877 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:35.341 INFO:teuthology.orchestra.run.smithi161.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 255 kB/s | 216 kB 00:00 2024-03-25T16:23:35.416 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:35.455 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:35.531 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:35.531 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:23:35.531 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:23:35.531 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:35.573 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:35.949 INFO:teuthology.orchestra.run.smithi161.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 57 kB/s | 35 kB 00:00 2024-03-25T16:23:36.583 INFO:teuthology.orchestra.run.smithi161.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 70 kB/s | 44 kB 00:00 2024-03-25T16:23:37.450 INFO:teuthology.orchestra.run.smithi161.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 357 kB/s | 309 kB 00:00 2024-03-25T16:23:41.416 INFO:teuthology.orchestra.run.smithi161.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 551 kB/s | 5.1 MB 00:09 2024-03-25T16:23:41.891 INFO:teuthology.orchestra.run.smithi161.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 55 kB/s | 26 kB 00:00 2024-03-25T16:23:42.083 INFO:teuthology.orchestra.run.smithi161.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 253 kB/s | 49 kB 00:00 2024-03-25T16:23:42.276 INFO:teuthology.orchestra.run.smithi161.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-03-25T16:23:43.267 INFO:teuthology.orchestra.run.smithi161.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 778 kB/s | 4.4 MB 00:05 2024-03-25T16:23:43.468 INFO:teuthology.orchestra.run.smithi161.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 703 kB/s | 838 kB 00:01 2024-03-25T16:23:43.568 INFO:teuthology.orchestra.run.smithi161.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 639 kB/s | 192 kB 00:00 2024-03-25T16:23:43.752 INFO:teuthology.orchestra.run.smithi161.stdout:(29/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 816 kB/s | 20 MB 00:24 2024-03-25T16:23:44.741 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:44.741 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2024-03-25T16:23:44.741 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2024-03-25T16:23:44.741 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2024-03-25T16:23:44.742 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2024-03-25T16:23:44.742 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2024-03-25T16:23:44.742 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /root 2024-03-25T16:23:44.742 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2024-03-25T16:23:44.742 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:45.601 INFO:teuthology.orchestra.run.smithi161.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 759 kB/s | 1.6 MB 00:02 2024-03-25T16:23:45.639 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:45.662 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:45.662 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:45.663 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:23:45.663 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:45.663 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:45.663 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:45.664 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:45.750 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:45.750 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:23:46.887 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:23:46.888 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:23:46.889 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:23:47.876 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:47.876 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:47.876 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2024-03-25T16:23:47.876 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.876 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.876 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.877 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:23:47.878 INFO:teuthology.orchestra.run.smithi132.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:23:47.879 INFO:teuthology.orchestra.run.smithi132.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:23:47.879 INFO:teuthology.orchestra.run.smithi132.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:23:47.879 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:47.879 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:23:48.130 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2024-03-25T16:23:48.689 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:56 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:23:48.839 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:23:48.840 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 48 M 2024-03-25T16:23:48.841 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 204 M 2024-03-25T16:23:48.842 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:23:49.122 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-25T16:23:49.222 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.4 MB/s | 300 kB 00:00 2024-03-25T16:23:49.248 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-25T16:23:49.326 INFO:teuthology.orchestra.run.smithi161.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 575 kB/s | 3.1 MB 00:05 2024-03-25T16:23:49.576 INFO:teuthology.orchestra.run.smithi161.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 575 kB/s | 3.4 MB 00:06 2024-03-25T16:23:49.582 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:49.582 INFO:teuthology.orchestra.run.smithi161.stdout:Total 1.9 MB/s | 58 MB 00:30 2024-03-25T16:23:49.615 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:23:49.655 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:23:49.655 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:23:49.932 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 172 kB/s | 163 kB 00:00 2024-03-25T16:23:50.090 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 222 kB/s | 187 kB 00:00 2024-03-25T16:23:50.144 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:23:50.145 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:23:51.605 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:23:51.706 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:23:51.739 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:23:51.764 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:23:51.787 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:51.803 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:51.964 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:52.017 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:52.336 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:23:52.399 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:52.474 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:52.561 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:23:52.584 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:52.646 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:52.675 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:23:52.719 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:23:52.786 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:52.819 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:52.881 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:52.908 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:23:53.017 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:23:53.066 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:23:53.100 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:23:53.133 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:23:53.162 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:23:53.194 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:23:53.504 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:23:53.533 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:53.622 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:53.654 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:23:53.697 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:23:54.153 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:23:54.177 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:54.248 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:54.316 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:23:54.346 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:23:56.480 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:56.526 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:57.082 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:57.120 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:57.219 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:57.219 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:23:57.219 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:23:57.219 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:23:57.255 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-03-25T16:24:06.338 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-03-25T16:24:06.339 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-03-25T16:24:06.339 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:24:07.212 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:24:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:24:07.236 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:24:07.236 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:24:07.236 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:24:07.236 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:24:07.236 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:24:07.238 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:24:07.323 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:24:07.323 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:24:08.467 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:24:08.468 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:24:08.469 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:24:08.469 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:24:08.470 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:24:08.470 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:24:08.470 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:24:08.470 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:24:08.470 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:24:08.470 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:24:09.544 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:24:09.545 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:24:09.546 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:24:09.547 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:24:09.547 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:24:09.547 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:24:09.547 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:24:09.888 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-test 2024-03-25T16:24:10.476 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:00:54 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:24:10.631 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:24:10.632 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:24:10.632 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:24:10.632 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:24:10.632 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout:Install 6 Packages 2024-03-25T16:24:10.633 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:24:10.634 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 48 M 2024-03-25T16:24:10.634 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 204 M 2024-03-25T16:24:10.634 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:24:11.091 INFO:teuthology.orchestra.run.smithi161.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T16:24:11.141 INFO:teuthology.orchestra.run.smithi161.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:24:11.174 INFO:teuthology.orchestra.run.smithi161.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:24:11.433 INFO:teuthology.orchestra.run.smithi161.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 724 kB/s | 187 kB 00:00 2024-03-25T16:24:11.816 INFO:teuthology.orchestra.run.smithi161.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 181 kB/s | 163 kB 00:00 2024-03-25T16:25:38.327 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 443 kB/s | 47 MB 01:49 2024-03-25T16:25:38.329 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:38.329 INFO:teuthology.orchestra.run.smithi132.stdout:Total 451 kB/s | 48 MB 01:49 2024-03-25T16:25:38.340 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:25:38.360 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:25:38.360 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:25:38.742 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:25:38.742 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:25:39.336 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:25:39.401 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:25:39.451 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:25:39.491 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:25:39.544 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:25:39.588 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:25:44.049 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:25:44.120 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:25:46.502 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:25:46.502 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:25:46.502 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:25:46.502 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:25:46.502 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:25:46.502 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:25:46.949 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:25:46.949 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:25:46.949 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:25:46.950 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:25:47.240 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2024-03-25T16:25:47.809 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:55 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:25:47.967 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:25:47.969 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-25T16:25:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:25:47.969 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-25T16:25:47.969 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:25:47.969 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:25:47.969 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:25:47.970 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:25:47.971 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout:========================================================================================= 2024-03-25T16:25:47.972 INFO:teuthology.orchestra.run.smithi132.stdout:Install 36 Packages 2024-03-25T16:25:47.973 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:25:47.973 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 30 M 2024-03-25T16:25:47.973 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 107 M 2024-03-25T16:25:47.973 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:25:48.922 INFO:teuthology.orchestra.run.smithi132.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:25:53.588 INFO:teuthology.orchestra.run.smithi132.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 297 kB/s | 1.5 MB 00:05 2024-03-25T16:25:55.122 INFO:teuthology.orchestra.run.smithi132.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 326 kB/s | 2.1 MB 00:06 2024-03-25T16:25:55.588 INFO:teuthology.orchestra.run.smithi132.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 530 kB/s | 247 kB 00:00 2024-03-25T16:25:55.880 INFO:teuthology.orchestra.run.smithi132.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 640 kB/s | 187 kB 00:00 2024-03-25T16:25:55.947 INFO:teuthology.orchestra.run.smithi132.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-25T16:25:55.998 INFO:teuthology.orchestra.run.smithi132.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2024-03-25T16:25:56.040 INFO:teuthology.orchestra.run.smithi132.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:25:56.081 INFO:teuthology.orchestra.run.smithi132.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 933 kB/s | 39 kB 00:00 2024-03-25T16:25:56.141 INFO:teuthology.orchestra.run.smithi132.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-25T16:25:56.216 INFO:teuthology.orchestra.run.smithi132.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:25:56.249 INFO:teuthology.orchestra.run.smithi132.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:25:56.283 INFO:teuthology.orchestra.run.smithi132.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:25:56.325 INFO:teuthology.orchestra.run.smithi132.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:25:56.875 INFO:teuthology.orchestra.run.smithi132.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 227 kB/s | 125 kB 00:00 2024-03-25T16:25:57.242 INFO:teuthology.orchestra.run.smithi132.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 61 kB/s | 22 kB 00:00 2024-03-25T16:25:57.626 INFO:teuthology.orchestra.run.smithi132.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-25T16:25:57.721 INFO:teuthology.orchestra.run.smithi132.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T16:25:57.860 INFO:teuthology.orchestra.run.smithi132.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T16:25:58.027 INFO:teuthology.orchestra.run.smithi132.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-25T16:25:58.110 INFO:teuthology.orchestra.run.smithi132.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T16:25:58.185 INFO:teuthology.orchestra.run.smithi132.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 235 kB/s | 18 kB 00:00 2024-03-25T16:25:58.261 INFO:teuthology.orchestra.run.smithi132.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 309 kB/s | 23 kB 00:00 2024-03-25T16:25:58.336 INFO:teuthology.orchestra.run.smithi132.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-25T16:25:58.411 INFO:teuthology.orchestra.run.smithi132.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-25T16:25:58.487 INFO:teuthology.orchestra.run.smithi132.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 618 kB/s | 46 kB 00:00 2024-03-25T16:25:58.570 INFO:teuthology.orchestra.run.smithi132.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 946 kB/s | 79 kB 00:00 2024-03-25T16:25:58.679 INFO:teuthology.orchestra.run.smithi132.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.5 MB/s | 272 kB 00:00 2024-03-25T16:25:58.754 INFO:teuthology.orchestra.run.smithi132.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2024-03-25T16:25:58.854 INFO:teuthology.orchestra.run.smithi132.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 901 kB/s | 90 kB 00:00 2024-03-25T16:25:58.930 INFO:teuthology.orchestra.run.smithi132.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 476 kB/s | 36 kB 00:00 2024-03-25T16:25:59.030 INFO:teuthology.orchestra.run.smithi132.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-25T16:25:59.164 INFO:teuthology.orchestra.run.smithi132.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-25T16:25:59.239 INFO:teuthology.orchestra.run.smithi132.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 266 kB/s | 20 kB 00:00 2024-03-25T16:25:59.997 INFO:teuthology.orchestra.run.smithi132.stdout:(35/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.6 MB/s | 17 MB 00:06 2024-03-25T16:26:02.447 INFO:teuthology.orchestra.run.smithi132.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 355 kB/s | 4.7 MB 00:13 2024-03-25T16:26:02.451 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:02.451 INFO:teuthology.orchestra.run.smithi132.stdout:Total 2.0 MB/s | 30 MB 00:14 2024-03-25T16:26:02.477 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:26:02.519 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:26:02.519 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:26:02.937 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:26:02.937 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:26:04.246 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:26:04.308 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:26:04.351 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:26:04.386 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:26:04.456 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:26:04.493 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:26:04.526 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:26:04.626 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:26:04.704 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:26:04.753 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:26:04.833 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:26:04.888 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:26:04.926 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:26:04.983 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:26:05.038 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:26:05.223 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:26:05.273 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:26:05.311 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:26:05.353 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:26:05.384 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:26:05.459 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:26:05.494 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:26:05.527 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:26:05.568 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:26:05.603 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:26:05.638 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:26:05.727 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:26:05.769 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:26:05.849 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:26:05.986 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:26:06.112 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:26:06.271 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:26:06.307 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:26:06.332 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:26:06.332 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:06.332 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:26:06.333 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:26:06.333 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:26:06.333 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:07.758 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:26:07.784 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:26:07.785 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:07.785 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:26:07.785 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:26:07.785 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:26:07.785 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:08.225 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:26:08.249 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:26:08.249 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:08.249 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:26:08.249 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:26:08.249 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:26:08.249 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:08.328 INFO:teuthology.orchestra.run.smithi161.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 412 kB/s | 47 MB 01:57 2024-03-25T16:26:08.330 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:08.334 INFO:teuthology.orchestra.run.smithi161.stdout:Total 419 kB/s | 48 MB 01:57 2024-03-25T16:26:08.341 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:26:08.361 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:26:08.361 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:26:08.468 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:26:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:26:08.491 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:08.491 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:26:08.491 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:26:08.491 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:26:08.491 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:08.566 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:26:08.742 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:26:08.742 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:26:09.334 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:26:09.420 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:26:09.468 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:26:09.508 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:26:09.559 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:26:09.589 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:26:10.353 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:26:10.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:26:10.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:26:10.353 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:26:10.354 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:26:10.355 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:26:10.357 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:26:10.358 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:26:10.358 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:26:10.358 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:26:11.413 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:26:11.413 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:11.413 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:26:11.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:11.413 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:26:11.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:26:11.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:11.416 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:26:11.811 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2024-03-25T16:26:12.374 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:20 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:26:12.471 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:12.523 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:26:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:26:12.524 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:26:12.590 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2024-03-25T16:26:13.149 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:21 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:26:13.297 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:26:13.298 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:26:13.298 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:26:13.298 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:26:13.298 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 747 k 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 754 k 2024-03-25T16:26:13.299 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:26:14.149 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:26:14.221 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:26:15.320 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 370 kB/s | 747 kB 00:02 2024-03-25T16:26:15.320 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:15.321 INFO:teuthology.orchestra.run.smithi132.stdout:Total 370 kB/s | 747 kB 00:02 2024-03-25T16:26:15.321 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:26:15.327 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:26:15.327 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:26:15.333 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:26:15.333 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:26:15.413 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:26:15.804 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:15.901 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:16.295 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:16.572 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:26:16.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:26:16.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:26:16.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:26:16.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:26:16.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:26:16.579 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:16.579 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:16.579 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:26:16.579 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:16.579 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:16.579 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:26:16.786 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:26:16.933 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:16.934 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:26:17.257 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph 2024-03-25T16:26:17.361 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:25 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:26:17.515 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:26:17.516 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:26:17.516 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:26:17.516 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 145 k 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 436 k 2024-03-25T16:26:17.517 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:26:17.827 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:01 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:26:17.990 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:26:17.993 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:26:17.994 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:26:17.995 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout:========================================================================================= 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout:Install 36 Packages 2024-03-25T16:26:17.996 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:17.997 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 30 M 2024-03-25T16:26:17.997 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 107 M 2024-03-25T16:26:17.997 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:26:18.630 INFO:teuthology.orchestra.run.smithi161.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:26:18.694 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 124 kB/s | 145 kB 00:01 2024-03-25T16:26:18.694 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:18.694 INFO:teuthology.orchestra.run.smithi132.stdout:Total 123 kB/s | 145 kB 00:01 2024-03-25T16:26:18.695 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:26:18.703 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:26:18.703 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:26:18.758 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:26:18.758 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:26:18.900 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:26:18.961 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:26:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:26:18.986 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:18.986 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:26:18.986 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:19.993 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:26:19.993 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:19.993 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:26:19.993 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:19.994 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:19.994 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:26:20.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2024-03-25T16:26:20.797 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:28 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:26:20.888 INFO:teuthology.orchestra.run.smithi161.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 801 kB/s | 2.1 MB 00:02 2024-03-25T16:26:20.893 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:20.945 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:26:20.946 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:26:20.946 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:26:21.013 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:26:21.088 INFO:teuthology.orchestra.run.smithi161.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 523 kB/s | 1.5 MB 00:02 2024-03-25T16:26:21.488 INFO:teuthology.orchestra.run.smithi161.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 619 kB/s | 247 kB 00:00 2024-03-25T16:26:21.571 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:26:21.655 INFO:teuthology.orchestra.run.smithi161.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-25T16:26:21.705 INFO:teuthology.orchestra.run.smithi161.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T16:26:21.720 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:26:21.721 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-25T16:26:21.721 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:26:21.722 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout:====================================================================================== 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-03-25T16:26:21.723 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:21.724 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.9 M 2024-03-25T16:26:21.724 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-03-25T16:26:21.724 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:26:21.764 INFO:teuthology.orchestra.run.smithi161.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 540 kB/s | 32 kB 00:00 2024-03-25T16:26:21.814 INFO:teuthology.orchestra.run.smithi161.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-25T16:26:21.865 INFO:teuthology.orchestra.run.smithi161.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-03-25T16:26:21.915 INFO:teuthology.orchestra.run.smithi161.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:26:22.023 INFO:teuthology.orchestra.run.smithi161.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-25T16:26:22.057 INFO:teuthology.orchestra.run.smithi161.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:26:22.099 INFO:teuthology.orchestra.run.smithi161.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:26:22.141 INFO:teuthology.orchestra.run.smithi161.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:26:22.347 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T16:26:22.448 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 58 kB/s | 31 kB 00:00 2024-03-25T16:26:22.506 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-25T16:26:22.532 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-25T16:26:22.566 INFO:teuthology.orchestra.run.smithi161.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 294 kB/s | 125 kB 00:00 2024-03-25T16:26:22.650 INFO:teuthology.orchestra.run.smithi161.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-25T16:26:22.682 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.8 MB/s | 863 kB 00:00 2024-03-25T16:26:22.907 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-25T16:26:23.116 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.8 MB/s | 2.0 MB 00:00 2024-03-25T16:26:23.116 INFO:teuthology.orchestra.run.smithi161.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 324 kB/s | 151 kB 00:00 2024-03-25T16:26:23.200 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T16:26:23.217 INFO:teuthology.orchestra.run.smithi161.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 435 kB/s | 43 kB 00:00 2024-03-25T16:26:23.275 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 410 kB/s | 31 kB 00:00 2024-03-25T16:26:23.383 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2024-03-25T16:26:23.409 INFO:teuthology.orchestra.run.smithi161.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 896 kB/s | 172 kB 00:00 2024-03-25T16:26:23.676 INFO:teuthology.orchestra.run.smithi161.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-03-25T16:26:23.760 INFO:teuthology.orchestra.run.smithi161.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T16:26:23.852 INFO:teuthology.orchestra.run.smithi161.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T16:26:23.943 INFO:teuthology.orchestra.run.smithi161.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 253 kB/s | 23 kB 00:00 2024-03-25T16:26:24.044 INFO:teuthology.orchestra.run.smithi161.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T16:26:24.136 INFO:teuthology.orchestra.run.smithi161.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T16:26:24.236 INFO:teuthology.orchestra.run.smithi161.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 464 kB/s | 46 kB 00:00 2024-03-25T16:26:24.336 INFO:teuthology.orchestra.run.smithi161.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 788 kB/s | 79 kB 00:00 2024-03-25T16:26:24.462 INFO:teuthology.orchestra.run.smithi161.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T16:26:24.578 INFO:teuthology.orchestra.run.smithi161.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 141 kB/s | 16 kB 00:00 2024-03-25T16:26:24.678 INFO:teuthology.orchestra.run.smithi161.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 901 kB/s | 90 kB 00:00 2024-03-25T16:26:24.779 INFO:teuthology.orchestra.run.smithi161.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T16:26:24.913 INFO:teuthology.orchestra.run.smithi161.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-25T16:26:25.021 INFO:teuthology.orchestra.run.smithi161.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.9 MB/s | 427 kB 00:00 2024-03-25T16:26:25.146 INFO:teuthology.orchestra.run.smithi161.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 160 kB/s | 20 kB 00:00 2024-03-25T16:26:25.808 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 468 kB/s | 1.8 MB 00:03 2024-03-25T16:26:29.613 INFO:teuthology.orchestra.run.smithi161.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 437 kB/s | 4.7 MB 00:10 2024-03-25T16:26:30.442 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): grpc-data-1.46.7-10.el9.noarch.rpm 2.5 kB/s | 19 kB 00:07 2024-03-25T16:26:30.445 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:30.445 INFO:teuthology.orchestra.run.smithi132.stdout:Total 805 kB/s | 6.9 MB 00:08 2024-03-25T16:26:30.466 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:26:30.484 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:26:30.484 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:26:30.659 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:26:30.659 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:26:31.256 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:26:31.333 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:26:31.369 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:26:31.416 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:26:31.454 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:26:31.553 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:26:31.681 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:26:31.776 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:26:31.826 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:26:31.854 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:26:31.885 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:26:32.132 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:26:32.161 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:26:32.927 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:26:33.292 INFO:teuthology.orchestra.run.smithi132.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:26:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:26:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:26:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:26:33.293 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:33.293 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:26:33.581 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:26:34.144 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:03:42 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:26:34.296 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:26:34.297 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-25T16:26:34.297 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:26:34.297 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-25T16:26:34.297 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:26:34.297 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:26:34.298 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout:============================================================================================ 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout:Install 12 Packages 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 37 M 2024-03-25T16:26:34.299 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 209 M 2024-03-25T16:26:34.300 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:26:34.690 INFO:teuthology.orchestra.run.smithi132.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 196 kB/s | 32 kB 00:00 2024-03-25T16:26:34.715 INFO:teuthology.orchestra.run.smithi132.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 700 kB/s | 17 kB 00:00 2024-03-25T16:26:34.757 INFO:teuthology.orchestra.run.smithi132.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 830 kB/s | 34 kB 00:00 2024-03-25T16:26:34.807 INFO:teuthology.orchestra.run.smithi132.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-25T16:26:35.033 INFO:teuthology.orchestra.run.smithi132.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-03-25T16:26:35.099 INFO:teuthology.orchestra.run.smithi132.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T16:26:35.125 INFO:teuthology.orchestra.run.smithi132.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:26:35.500 INFO:teuthology.orchestra.run.smithi132.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-25T16:26:35.859 INFO:teuthology.orchestra.run.smithi132.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 545 kB/s | 195 kB 00:00 2024-03-25T16:26:36.026 INFO:teuthology.orchestra.run.smithi132.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-25T16:26:36.159 INFO:teuthology.orchestra.run.smithi132.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 777 kB/s | 803 kB 00:01 2024-03-25T16:26:51.927 INFO:teuthology.orchestra.run.smithi161.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 553 kB/s | 17 MB 00:31 2024-03-25T16:26:51.930 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:51.930 INFO:teuthology.orchestra.run.smithi161.stdout:Total 895 kB/s | 30 MB 00:33 2024-03-25T16:26:51.955 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:26:51.998 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:26:51.998 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:26:52.407 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:26:52.407 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:26:53.724 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:26:53.788 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:26:53.830 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:26:53.856 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:26:53.925 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:26:53.963 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:26:53.996 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:26:54.088 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:26:54.165 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:26:54.207 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:26:54.277 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:26:54.324 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:26:54.362 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:26:54.411 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:26:54.466 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:26:54.650 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:26:54.700 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:26:54.730 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:26:54.765 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:26:54.795 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:26:54.871 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:26:54.905 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:26:54.939 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:26:54.978 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:26:55.014 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:26:55.048 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:26:55.137 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:26:55.179 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:26:55.251 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:26:55.387 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:26:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:26:55.671 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:26:55.701 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:26:55.725 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:26:55.725 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:55.725 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:26:55.725 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:26:55.725 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:26:55.725 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:57.168 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:26:57.195 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:26:57.195 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:57.195 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:26:57.195 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:26:57.195 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:26:57.195 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:57.432 INFO:teuthology.orchestra.run.smithi132.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 330 kB/s | 7.4 MB 00:22 2024-03-25T16:26:57.434 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:57.434 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.6 MB/s | 37 MB 00:23 2024-03-25T16:26:57.478 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:26:57.503 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:26:57.503 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:26:57.635 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:26:57.659 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:26:57.659 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:57.659 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:26:57.659 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:26:57.659 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:26:57.659 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:57.859 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:26:57.859 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:26:57.869 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:26:57.892 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:26:57.892 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:57.892 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:26:57.892 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:26:57.892 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:26:57.892 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:26:57.958 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:26:58.586 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:26:58.663 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:26:58.702 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:26:58.804 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:58.847 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:26:59.026 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:59.054 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:59.136 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:26:59.569 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:26:59.569 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:26:59.569 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:26:59.570 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:26:59.571 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:26:59.572 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:26:59.779 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:26:59.892 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:00.485 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:27:00.486 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:27:00.487 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:00.488 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:00.832 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-base 2024-03-25T16:27:01.123 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:27:01.398 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:45 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:01.497 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:01.548 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:01.549 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:27:01.549 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:01.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install cephadm 2024-03-25T16:27:02.145 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:27:02.174 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:46 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:02.180 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:02.324 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:27:02.325 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:27:02.326 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:02.326 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 747 k 2024-03-25T16:27:02.326 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 754 k 2024-03-25T16:27:02.326 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:27:04.408 INFO:teuthology.orchestra.run.smithi161.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 359 kB/s | 747 kB 00:02 2024-03-25T16:27:04.409 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:04.409 INFO:teuthology.orchestra.run.smithi161.stdout:Total 359 kB/s | 747 kB 00:02 2024-03-25T16:27:04.409 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:27:04.415 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:27:04.416 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:27:04.422 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:27:04.422 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:27:04.501 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:27:04.851 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:27:04.852 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:27:04.852 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:27:04.968 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:05.055 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:27:05.258 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:05.258 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:05.259 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:05.464 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:05.688 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:27:05.737 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:27:05.738 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:05.738 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:27:05.738 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:05.738 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:05.738 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:05.937 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:27:06.262 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:06.412 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:06.413 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:27:06.414 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2024-03-25T16:27:06.415 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:06.416 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 2.0 M 2024-03-25T16:27:06.416 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 26 M 2024-03-25T16:27:06.416 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:06.518 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:50 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:06.647 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 24 kB 00:00 2024-03-25T16:27:06.677 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:27:06.678 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:27:06.679 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:06.679 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 145 k 2024-03-25T16:27:06.679 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 436 k 2024-03-25T16:27:06.679 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:27:06.689 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 90 kB/s | 18 kB 00:00 2024-03-25T16:27:06.756 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-25T16:27:06.781 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-25T16:27:06.807 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:27:06.832 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 851 kB/s | 43 kB 00:00 2024-03-25T16:27:07.126 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 82 kB/s | 52 kB 00:00 2024-03-25T16:27:07.174 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2024-03-25T16:27:07.241 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 74 kB/s | 32 kB 00:00 2024-03-25T16:27:07.425 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-03-25T16:27:07.567 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 633 kB/s | 90 kB 00:00 2024-03-25T16:27:07.683 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-25T16:27:07.709 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-google-auth-2.28.2-1.el9.noarc 347 kB/s | 203 kB 00:00 2024-03-25T16:27:07.713 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:07.714 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T16:27:07.714 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:07.727 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:07.727 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:07.819 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:07.820 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:07.872 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 122 kB/s | 145 kB 00:01 2024-03-25T16:27:07.872 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:07.873 INFO:teuthology.orchestra.run.smithi161.stdout:Total 122 kB/s | 145 kB 00:01 2024-03-25T16:27:07.873 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:27:07.882 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:27:07.882 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:27:07.938 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:27:07.939 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:27:08.084 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:27:08.173 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:27:08.199 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:27:08.199 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:08.199 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:27:08.199 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:08.441 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:08.622 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:27:08.686 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:27:08.741 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:27:08.778 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:27:08.819 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:27:08.930 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:27:09.048 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:27:09.104 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:27:09.163 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:27:09.163 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:09.163 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:27:09.163 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:09.163 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:09.164 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:09.435 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr 2024-03-25T16:27:09.828 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:27:09.878 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:27:09.912 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:27:09.960 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:27:09.982 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:27:10.010 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:10.110 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:10.163 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:10.164 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:27:10.164 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:10.227 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:27:10.691 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:27:10.692 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:27:10.692 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:27:10.692 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:27:10.692 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:27:10.692 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:27:10.692 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:27:10.789 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:10.938 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:10.939 INFO:teuthology.orchestra.run.smithi161.stdout:====================================================================================== 2024-03-25T16:27:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:27:10.939 INFO:teuthology.orchestra.run.smithi161.stdout:====================================================================================== 2024-03-25T16:27:10.939 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:27:10.939 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:10.940 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:27:10.941 INFO:teuthology.orchestra.run.smithi161.stdout:====================================================================================== 2024-03-25T16:27:10.941 INFO:teuthology.orchestra.run.smithi161.stdout:Install 12 Packages 2024-03-25T16:27:10.941 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:10.941 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 6.9 M 2024-03-25T16:27:10.941 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 26 M 2024-03-25T16:27:10.941 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:27:11.063 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:27:11.063 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:11.063 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:27:11.064 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:27:11.065 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:11.065 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:11.329 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:27:11.650 INFO:teuthology.orchestra.run.smithi161.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T16:27:11.775 INFO:teuthology.orchestra.run.smithi161.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 45 kB/s | 31 kB 00:00 2024-03-25T16:27:11.818 INFO:teuthology.orchestra.run.smithi161.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-25T16:27:11.843 INFO:teuthology.orchestra.run.smithi161.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.5 MB/s | 238 kB 00:00 2024-03-25T16:27:11.892 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:11.960 INFO:teuthology.orchestra.run.smithi161.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-03-25T16:27:12.010 INFO:teuthology.orchestra.run.smithi161.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 391 kB/s | 19 kB 00:00 2024-03-25T16:27:12.042 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:27:12.043 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2024-03-25T16:27:12.044 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:12.045 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 6.8 M 2024-03-25T16:27:12.045 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 33 M 2024-03-25T16:27:12.045 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:12.093 INFO:teuthology.orchestra.run.smithi161.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 24 MB/s | 2.0 MB 00:00 2024-03-25T16:27:12.119 INFO:teuthology.orchestra.run.smithi161.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.8 MB/s | 863 kB 00:00 2024-03-25T16:27:12.144 INFO:teuthology.orchestra.run.smithi161.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-25T16:27:12.177 INFO:teuthology.orchestra.run.smithi161.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 5.5 MB/s | 188 kB 00:00 2024-03-25T16:27:12.228 INFO:teuthology.orchestra.run.smithi161.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-25T16:27:12.538 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-25T16:27:12.563 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 756 kB/s | 248 kB 00:00 2024-03-25T16:27:12.664 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 577 kB/s | 58 kB 00:00 2024-03-25T16:27:12.714 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-25T16:27:12.839 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:27:12.964 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 189 kB/s | 138 kB 00:00 2024-03-25T16:27:12.965 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:12.965 INFO:teuthology.orchestra.run.smithi132.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2024-03-25T16:27:12.978 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:12.989 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:12.989 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:13.077 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:13.078 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:13.373 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:13.553 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:27:13.599 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:27:14.145 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:27:14.227 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:27:14.288 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:27:14.311 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:27:15.160 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:27:15.160 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:27:15.160 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:27:15.160 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:27:15.160 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:27:15.160 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:27:15.478 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:27:15.479 INFO:teuthology.orchestra.run.smithi132.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:27:15.479 INFO:teuthology.orchestra.run.smithi132.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:27:15.479 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:15.479 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:15.728 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2024-03-25T16:27:16.294 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:16.443 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-25T16:27:16.444 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:16.445 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 964 k 2024-03-25T16:27:16.445 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.8 M 2024-03-25T16:27:16.445 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:16.591 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 538 kB/s | 79 kB 00:00 2024-03-25T16:27:18.266 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 486 kB/s | 885 kB 00:01 2024-03-25T16:27:18.268 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:18.268 INFO:teuthology.orchestra.run.smithi132.stdout:Total 528 kB/s | 964 kB 00:01 2024-03-25T16:27:18.269 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:18.292 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:18.292 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:18.369 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:18.370 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:18.549 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:18.692 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:19.430 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:19.431 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:27:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:27:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:19.724 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:27:19.724 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:19.724 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:20.027 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-volume 2024-03-25T16:27:20.435 INFO:teuthology.orchestra.run.smithi161.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 195 kB/s | 1.8 MB 00:09 2024-03-25T16:27:20.439 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:20.439 INFO:teuthology.orchestra.run.smithi161.stdout:Total 739 kB/s | 6.9 MB 00:09 2024-03-25T16:27:20.458 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:27:20.478 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:27:20.478 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:27:20.597 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:28 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:20.658 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:27:20.658 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:27:20.751 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:20.752 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:20.752 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2024-03-25T16:27:20.752 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 332 k 2024-03-25T16:27:20.753 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.5 M 2024-03-25T16:27:20.754 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:20.889 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 509 kB/s | 69 kB 00:00 2024-03-25T16:27:21.189 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:27:21.270 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:27:21.304 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:27:21.352 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:27:21.382 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:27:21.482 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:27:21.608 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:27:21.701 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:27:21.753 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:27:21.781 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:27:21.812 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:27:22.056 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 203 kB/s | 264 kB 00:01 2024-03-25T16:27:22.057 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:27:22.057 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:22.058 INFO:teuthology.orchestra.run.smithi132.stdout:Total 255 kB/s | 332 kB 00:01 2024-03-25T16:27:22.058 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:22.067 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:22.067 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:22.096 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:27:22.112 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:22.112 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:22.262 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:22.446 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:27:22.483 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:27:22.499 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:27:22.500 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:22.500 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:27:22.500 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:27:22.884 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:27:22.885 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:27:23.225 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:27:23.259 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:23.260 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:23.261 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:27:23.505 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:23.594 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:23.957 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2024-03-25T16:27:24.068 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:24.219 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:27:24.220 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout:Install 12 Packages 2024-03-25T16:27:24.221 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:24.222 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 37 M 2024-03-25T16:27:24.222 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 209 M 2024-03-25T16:27:24.222 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:27:24.541 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:32 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:24.650 INFO:teuthology.orchestra.run.smithi161.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 282 kB/s | 32 kB 00:00 2024-03-25T16:27:24.690 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:27:24.691 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:24.692 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 127 k 2024-03-25T16:27:24.692 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 456 k 2024-03-25T16:27:24.692 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:24.708 INFO:teuthology.orchestra.run.smithi161.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 491 kB/s | 17 kB 00:00 2024-03-25T16:27:24.742 INFO:teuthology.orchestra.run.smithi161.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-25T16:27:24.909 INFO:teuthology.orchestra.run.smithi161.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-25T16:27:25.076 INFO:teuthology.orchestra.run.smithi161.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T16:27:25.135 INFO:teuthology.orchestra.run.smithi161.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.0 MB/s | 3.0 MB 00:00 2024-03-25T16:27:25.174 INFO:teuthology.orchestra.run.smithi161.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-25T16:27:25.468 INFO:teuthology.orchestra.run.smithi161.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-25T16:27:25.495 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 158 kB/s | 127 kB 00:00 2024-03-25T16:27:25.496 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:25.496 INFO:teuthology.orchestra.run.smithi132.stdout:Total 157 kB/s | 127 kB 00:00 2024-03-25T16:27:25.496 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:25.502 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:25.502 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:25.559 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:25.559 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:25.702 INFO:teuthology.orchestra.run.smithi161.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 838 kB/s | 195 kB 00:00 2024-03-25T16:27:25.707 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:25.727 INFO:teuthology.orchestra.run.smithi161.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-25T16:27:25.810 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:26.327 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:26.626 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:26.626 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:26.626 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:26.626 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:26.627 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:26.627 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:26.635 INFO:teuthology.orchestra.run.smithi161.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 263 kB/s | 245 kB 00:00 2024-03-25T16:27:26.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2024-03-25T16:27:27.418 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:27.518 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:27.573 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:27.574 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:27.643 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2024-03-25T16:27:28.216 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:36 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:28.366 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:28.367 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:28.367 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:27:28.367 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:28.367 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:28.367 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 k 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 140 k 2024-03-25T16:27:28.368 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:28.824 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 68 kB/s | 31 kB 00:00 2024-03-25T16:27:28.825 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:28.825 INFO:teuthology.orchestra.run.smithi132.stdout:Total 68 kB/s | 31 kB 00:00 2024-03-25T16:27:28.825 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:28.829 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:28.829 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:28.870 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:28.870 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:28.995 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:29.081 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:29.506 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:29.774 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:29.774 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:29.774 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:29.774 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:29.775 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:29.775 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:29.947 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2024-03-25T16:27:30.519 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:30.616 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:30.667 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:30.668 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:30.668 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:30.719 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2024-03-25T16:27:31.282 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:39 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:31.378 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:31.430 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:31.431 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:31.431 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:31.477 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2024-03-25T16:27:32.041 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:32.137 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:32.189 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:32.190 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:32.190 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:32.235 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2024-03-25T16:27:32.801 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:32.898 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:32.949 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:32.950 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:32.950 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:33.006 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2024-03-25T16:27:33.606 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:33.709 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:33.763 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:33.764 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:33.764 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:33.824 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2024-03-25T16:27:34.395 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:34.491 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:34.542 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:34.543 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T16:27:34.543 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:34.596 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2024-03-25T16:27:35.161 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:35.310 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:35.311 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:35.312 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:27:35.312 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:35.312 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 85 k 2024-03-25T16:27:35.312 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 231 k 2024-03-25T16:27:35.312 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:35.351 INFO:teuthology.orchestra.run.smithi161.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 700 kB/s | 7.4 MB 00:10 2024-03-25T16:27:35.353 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:35.354 INFO:teuthology.orchestra.run.smithi161.stdout:Total 3.3 MB/s | 37 MB 00:11 2024-03-25T16:27:35.397 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:27:35.423 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:27:35.424 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:27:35.783 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:27:35.783 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:27:36.078 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 111 kB/s | 85 kB 00:00 2024-03-25T16:27:36.079 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:36.079 INFO:teuthology.orchestra.run.smithi132.stdout:Total 111 kB/s | 85 kB 00:00 2024-03-25T16:27:36.079 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:36.086 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:36.086 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:36.143 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:36.143 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:36.284 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:36.393 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:36.495 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:27:36.574 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:27:36.597 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:27:36.689 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:27:36.725 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:27:36.833 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:36.896 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:27:36.916 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:27:36.999 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:27:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:37.129 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:37.129 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:37.129 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:37.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2024-03-25T16:27:37.647 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:27:37.770 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:27:37.938 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:38.098 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:38.099 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:38.100 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:38.100 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:27:38.100 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:38.100 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.1 M 2024-03-25T16:27:38.100 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 13 M 2024-03-25T16:27:38.100 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:39.098 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:27:40.120 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:27:40.148 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:42.582 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:42.582 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:27:42.583 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:27:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:27:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:42.992 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:27:42.992 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:42.993 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:43.405 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:27:43.973 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:44.124 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:27:44.125 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:27:44.126 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:27:44.127 INFO:teuthology.orchestra.run.smithi161.stdout:Install 13 Packages 2024-03-25T16:27:44.127 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:44.127 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 2.0 M 2024-03-25T16:27:44.127 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 26 M 2024-03-25T16:27:44.127 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:27:44.367 INFO:teuthology.orchestra.run.smithi161.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 187 kB/s | 18 kB 00:00 2024-03-25T16:27:44.417 INFO:teuthology.orchestra.run.smithi161.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-03-25T16:27:44.467 INFO:teuthology.orchestra.run.smithi161.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T16:27:44.492 INFO:teuthology.orchestra.run.smithi161.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:27:44.518 INFO:teuthology.orchestra.run.smithi161.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:27:44.543 INFO:teuthology.orchestra.run.smithi161.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 866 kB/s | 43 kB 00:00 2024-03-25T16:27:44.826 INFO:teuthology.orchestra.run.smithi161.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 94 kB/s | 52 kB 00:00 2024-03-25T16:27:45.010 INFO:teuthology.orchestra.run.smithi161.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-03-25T16:27:45.035 INFO:teuthology.orchestra.run.smithi161.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 30 kB/s | 15 kB 00:00 2024-03-25T16:27:45.219 INFO:teuthology.orchestra.run.smithi161.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 325 kB/s | 59 kB 00:00 2024-03-25T16:27:45.394 INFO:teuthology.orchestra.run.smithi161.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 512 kB/s | 90 kB 00:00 2024-03-25T16:27:45.545 INFO:teuthology.orchestra.run.smithi161.stdout:(12/13): python3-google-auth-2.28.2-1.el9.noarc 282 kB/s | 203 kB 00:00 2024-03-25T16:27:45.789 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 411 kB/s | 3.1 MB 00:07 2024-03-25T16:27:45.790 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:45.791 INFO:teuthology.orchestra.run.smithi132.stdout:Total 411 kB/s | 3.1 MB 00:07 2024-03-25T16:27:45.791 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:45.801 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:45.801 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:45.877 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:45.878 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:46.309 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:46.364 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:46.389 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:46.389 INFO:teuthology.orchestra.run.smithi132.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:46.389 INFO:teuthology.orchestra.run.smithi132.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:27:46.389 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:46.389 INFO:teuthology.orchestra.run.smithi132.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:46.389 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:46.470 INFO:teuthology.orchestra.run.smithi161.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 716 kB/s | 1.0 MB 00:01 2024-03-25T16:27:46.474 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:46.475 INFO:teuthology.orchestra.run.smithi161.stdout:Total 893 kB/s | 2.0 MB 00:02 2024-03-25T16:27:46.475 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:27:46.490 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:27:46.491 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:27:46.591 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:27:46.592 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:27:47.108 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:27:47.264 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:27:47.321 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:27:47.376 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:27:47.413 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:27:47.455 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:27:47.547 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:27:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:47.563 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:47.563 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:47.563 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:47.657 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:27:47.714 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:27:47.846 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2024-03-25T16:27:48.433 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:27:48.437 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:56 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:48.478 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:27:48.511 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:27:48.560 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:27:48.620 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:27:48.620 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:48.620 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 171 k 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 494 k 2024-03-25T16:27:48.621 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:49.400 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:27:49.400 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:27:49.400 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:27:49.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:27:49.484 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 192 kB/s | 171 kB 00:00 2024-03-25T16:27:49.485 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:49.485 INFO:teuthology.orchestra.run.smithi132.stdout:Total 191 kB/s | 171 kB 00:00 2024-03-25T16:27:49.485 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:27:49.494 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:27:49.494 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:27:49.553 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:27:49.553 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:27:49.702 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:27:49.758 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:27:49.759 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:27:49.760 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:27:49.760 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:27:49.760 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:49.760 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:27:49.816 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:50.036 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:27:50.273 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:50.562 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:50.562 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:50.562 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:27:50.562 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:50.563 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:50.563 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:27:50.598 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:27:50.747 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:27:50.748 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout:Install 6 Packages 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:27:50.749 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 6.8 M 2024-03-25T16:27:50.750 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 33 M 2024-03-25T16:27:50.750 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:27:50.792 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:27:51.095 INFO:teuthology.orchestra.run.smithi161.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.0 MB/s | 248 kB 00:00 2024-03-25T16:27:51.367 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:27:51.395 INFO:teuthology.orchestra.run.smithi161.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-25T16:27:51.521 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:27:51.521 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:51.521 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:27:51.521 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2024-03-25T16:27:51.522 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:27:51.523 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 91 M 2024-03-25T16:27:51.523 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 498 M 2024-03-25T16:27:51.523 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:27:51.888 INFO:teuthology.orchestra.run.smithi161.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 151 kB/s | 138 kB 00:00 2024-03-25T16:27:52.680 INFO:teuthology.orchestra.run.smithi161.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 45 kB/s | 58 kB 00:01 2024-03-25T16:27:54.554 INFO:teuthology.orchestra.run.smithi161.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 160 kB/s | 548 kB 00:03 2024-03-25T16:28:07.711 INFO:teuthology.orchestra.run.smithi161.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 3.2 kB/s | 51 kB 00:15 2024-03-25T16:28:07.712 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:07.712 INFO:teuthology.orchestra.run.smithi161.stdout:Total 410 kB/s | 6.8 MB 00:16 2024-03-25T16:28:07.727 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:07.739 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:07.740 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:07.832 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:07.833 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:08.132 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:08.293 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:28:08.354 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:28:08.863 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:28:08.932 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:28:09.001 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:28:09.025 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:28:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:28:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:28:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:28:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:28:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:28:09.798 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:10.094 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:10.329 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-fuse 2024-03-25T16:28:10.929 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:11.088 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:11.089 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:11.089 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:28:11.089 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:11.089 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 964 k 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 2.8 M 2024-03-25T16:28:11.090 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:11.204 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 694 kB/s | 79 kB 00:00 2024-03-25T16:28:13.021 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 459 kB/s | 885 kB 00:01 2024-03-25T16:28:13.023 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:13.023 INFO:teuthology.orchestra.run.smithi161.stdout:Total 498 kB/s | 964 kB 00:01 2024-03-25T16:28:13.023 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:13.047 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:13.047 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:13.125 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:13.125 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:13.310 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:13.464 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:28:13.529 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:28:14.244 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:28:14.245 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:28:14.560 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:28:14.560 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:14.560 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:14.560 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:28:14.560 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:14.560 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:14.853 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-volume 2024-03-25T16:28:15.432 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:15.585 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:15.586 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:15.586 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-25T16:28:15.586 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:15.586 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:15.586 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2024-03-25T16:28:15.587 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:15.588 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 332 k 2024-03-25T16:28:15.588 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.5 M 2024-03-25T16:28:15.588 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:15.785 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 349 kB/s | 69 kB 00:00 2024-03-25T16:28:16.719 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 233 kB/s | 264 kB 00:01 2024-03-25T16:28:16.720 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:16.721 INFO:teuthology.orchestra.run.smithi161.stdout:Total 293 kB/s | 332 kB 00:01 2024-03-25T16:28:16.721 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:16.731 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:16.731 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:16.776 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:16.777 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:16.924 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:17.101 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:28:17.138 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:28:17.153 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:28:17.153 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:17.153 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:28:17.153 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:17.904 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:28:18.266 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:28:18.266 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:18.266 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:18.266 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:18.266 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:28:18.266 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:18.267 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:18.610 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados-devel 2024-03-25T16:28:19.185 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:19.335 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 127 k 2024-03-25T16:28:19.336 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 456 k 2024-03-25T16:28:19.337 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:20.582 INFO:teuthology.orchestra.run.smithi161.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 102 kB/s | 127 kB 00:01 2024-03-25T16:28:20.583 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:20.583 INFO:teuthology.orchestra.run.smithi161.stdout:Total 101 kB/s | 127 kB 00:01 2024-03-25T16:28:20.583 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:20.591 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:20.591 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:20.651 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:20.651 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:20.802 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:20.905 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:21.355 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:21.651 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:21.651 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:21.651 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:21.651 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:21.651 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:21.651 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:21.884 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs2 2024-03-25T16:28:22.463 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:22.565 INFO:teuthology.orchestra.run.smithi161.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:22.620 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:22.621 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:22.621 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:22.677 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs-devel 2024-03-25T16:28:23.249 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:23.400 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:23.401 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:23.401 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:28:23.401 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:23.401 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:23.401 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 31 k 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 140 k 2024-03-25T16:28:23.402 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:24.119 INFO:teuthology.orchestra.run.smithi161.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 43 kB/s | 31 kB 00:00 2024-03-25T16:28:24.119 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:24.119 INFO:teuthology.orchestra.run.smithi161.stdout:Total 43 kB/s | 31 kB 00:00 2024-03-25T16:28:24.120 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:24.123 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:24.123 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:24.152 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:24.153 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:24.264 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:24.375 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:24.720 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:25.177 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados2 2024-03-25T16:28:25.752 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:25.849 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:25.900 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:25.901 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:25.901 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:25.959 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librbd1 2024-03-25T16:28:26.537 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:26.635 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:26.687 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:26.688 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:26.689 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:26.739 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rados 2024-03-25T16:28:27.311 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:27.367 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 641 kB/s | 22 MB 00:35 2024-03-25T16:28:27.407 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:27.459 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:27.460 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:27.460 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:27.540 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rgw 2024-03-25T16:28:28.106 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:28.198 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:28.250 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:28.251 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:28.251 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:28.303 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-cephfs 2024-03-25T16:28:28.869 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:28.964 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:29.018 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:29.018 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:29.019 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:29.070 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rbd 2024-03-25T16:28:29.639 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:29.736 INFO:teuthology.orchestra.run.smithi161.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:29.788 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:29.789 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T16:28:29.789 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:29.847 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-fuse 2024-03-25T16:28:30.416 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:30.568 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:30.568 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 85 k 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 231 k 2024-03-25T16:28:30.569 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:31.296 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 117 kB/s | 85 kB 00:00 2024-03-25T16:28:31.297 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:31.298 INFO:teuthology.orchestra.run.smithi161.stdout:Total 117 kB/s | 85 kB 00:00 2024-03-25T16:28:31.298 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:31.305 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:31.305 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:31.361 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:31.361 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:31.494 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:31.594 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:31.991 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:32.293 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:32.293 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:32.293 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:32.293 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:32.293 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:32.294 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:32.532 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-mirror 2024-03-25T16:28:33.132 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:33.286 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:28:33.287 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:33.288 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 3.1 M 2024-03-25T16:28:33.288 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 13 M 2024-03-25T16:28:33.288 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:42.031 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 362 kB/s | 3.1 MB 00:08 2024-03-25T16:28:42.032 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:42.033 INFO:teuthology.orchestra.run.smithi161.stdout:Total 362 kB/s | 3.1 MB 00:08 2024-03-25T16:28:42.033 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:42.043 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:42.044 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:42.120 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:42.121 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:42.538 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:42.590 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:28:42.614 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:28:42.615 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:43.713 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:43.713 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:43.713 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:43.713 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:43.713 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:43.713 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:44.000 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-nbd 2024-03-25T16:28:44.616 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:28 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:44.773 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:44.773 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:44.773 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 171 k 2024-03-25T16:28:44.774 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 494 k 2024-03-25T16:28:44.775 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:28:46.201 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 120 kB/s | 171 kB 00:01 2024-03-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi161.stdout:Total 120 kB/s | 171 kB 00:01 2024-03-25T16:28:46.202 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:28:46.212 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:28:46.212 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:28:46.272 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:28:46.272 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:28:46.423 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:28:46.532 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:46.955 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:28:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:47.249 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:28:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:47.249 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:47.249 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:28:47.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:28:48.047 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:05:32 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:28:48.202 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:28:48.203 INFO:teuthology.orchestra.run.smithi161.stdout:Install 2 Packages 2024-03-25T16:28:48.203 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:28:48.203 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 91 M 2024-03-25T16:28:48.203 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 498 M 2024-03-25T16:28:48.203 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:29:53.614 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 571 kB/s | 68 MB 02:02 2024-03-25T16:29:53.615 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:53.615 INFO:teuthology.orchestra.run.smithi132.stdout:Total 760 kB/s | 91 MB 02:02 2024-03-25T16:29:53.616 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:29:53.620 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:29:53.620 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:29:54.208 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:29:54.208 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:29:57.321 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:29:57.486 INFO:teuthology.orchestra.run.smithi161.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 332 kB/s | 22 MB 01:09 2024-03-25T16:30:04.510 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:30:04.598 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:30:07.940 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:30:07.940 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:30:08.210 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:30:08.210 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:30:08.210 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:30:08.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:08.211 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:08.211 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:30:08.211 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:30:08.409 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:30:08.973 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:07:16 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:30:09.121 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:30:09.122 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:30:09.122 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 416 M 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.7 G 2024-03-25T16:30:09.123 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:32:01.562 INFO:teuthology.orchestra.run.smithi161.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 361 kB/s | 68 MB 03:13 2024-03-25T16:32:01.563 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:01.563 INFO:teuthology.orchestra.run.smithi161.stdout:Total 480 kB/s | 91 MB 03:13 2024-03-25T16:32:01.564 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:32:01.568 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:32:01.568 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:32:02.160 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:32:02.160 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:32:05.257 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:32:12.428 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:32:12.596 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:32:15.808 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:32:15.808 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:32:16.074 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:32:16.075 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:32:16.075 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:32:16.075 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:16.075 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:16.075 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:32:16.075 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:32:16.306 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:32:16.873 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:09:00 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:32:17.023 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:32:17.024 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:32:17.025 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:32:17.025 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:32:17.025 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:32:17.025 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 416 M 2024-03-25T16:32:17.025 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.7 G 2024-03-25T16:32:17.025 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:54:34.931 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 291 kB/s | 416 MB 24:25 2024-03-25T16:54:34.952 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:54:34.952 INFO:teuthology.orchestra.run.smithi132.stdout:Total 291 kB/s | 416 MB 24:25 2024-03-25T16:54:34.952 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:54:34.952 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:54:34.953 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:54:37.402 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:54:37.402 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:55:19.463 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:55:19.993 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:55:28.647 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:55:28.902 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:55:28.902 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:28.902 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:55:28.902 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:55:28.902 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:28.902 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:55:29.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:55:29.731 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:32:37 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:55:29.888 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:55:29.888 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:======================================================================================== 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.4 M 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 6.4 M 2024-03-25T16:55:29.889 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:55:32.100 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 305 kB/s | 416 MB 23:15 2024-03-25T16:55:32.100 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:32.100 INFO:teuthology.orchestra.run.smithi161.stdout:Total 305 kB/s | 416 MB 23:15 2024-03-25T16:55:32.101 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:55:32.104 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:55:32.104 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:55:34.568 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:55:34.568 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:55:35.619 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 255 kB/s | 1.4 MB 00:05 2024-03-25T16:55:35.620 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:35.621 INFO:teuthology.orchestra.run.smithi132.stdout:Total 255 kB/s | 1.4 MB 00:05 2024-03-25T16:55:35.621 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T16:55:35.626 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T16:55:35.626 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T16:55:35.670 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T16:55:35.670 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T16:55:35.927 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T16:55:36.034 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:55:36.438 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:55:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:55:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:36.739 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T16:55:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:55:36.739 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:36.739 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T16:55:36.945 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:55:37.509 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:32:45 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T16:55:37.657 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T16:55:37.657 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:55:37.657 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T16:55:37.657 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T16:55:37.658 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 176 M 2024-03-25T16:55:37.659 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 753 M 2024-03-25T16:55:37.659 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T16:56:16.735 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:56:17.095 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:25.504 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:25.761 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:56:25.761 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:25.761 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:56:25.762 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:25.762 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:25.762 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:56:26.007 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:56:26.603 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:33:10 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:56:26.762 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:56:26.762 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:56:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:56:26.762 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:56:26.762 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:56:26.762 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout:======================================================================================== 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 1.4 M 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 6.4 M 2024-03-25T16:56:26.763 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T16:56:31.323 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 321 kB/s | 1.4 MB 00:04 2024-03-25T16:56:31.323 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:56:31.324 INFO:teuthology.orchestra.run.smithi161.stdout:Total 320 kB/s | 1.4 MB 00:04 2024-03-25T16:56:31.324 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T16:56:31.328 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T16:56:31.328 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T16:56:31.373 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T16:56:31.373 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T16:56:31.627 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T16:56:31.737 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:56:32.121 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:56:32.401 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:56:32.401 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:32.401 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T16:56:32.401 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:56:32.401 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:32.401 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T16:56:32.586 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:56:33.148 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:33:17 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T16:56:33.296 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T16:56:33.296 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T16:56:33.297 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T16:56:33.298 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 176 M 2024-03-25T16:56:33.298 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 753 M 2024-03-25T16:56:33.298 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T17:06:35.844 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 273 kB/s | 176 MB 10:58 2024-03-25T17:06:35.844 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:06:35.844 INFO:teuthology.orchestra.run.smithi132.stdout:Total 273 kB/s | 176 MB 10:58 2024-03-25T17:06:35.844 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T17:06:35.847 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T17:06:35.847 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T17:06:36.908 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T17:06:36.909 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T17:06:55.136 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T17:06:55.497 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:07:00.276 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:07:00.540 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:07:00.541 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:07:00.541 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T17:07:00.541 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:07:00.541 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:07:00.541 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T17:07:00.771 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:07:01.337 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:44:09 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T17:07:01.487 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T17:07:01.488 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:07:01.488 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T17:07:01.488 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:07:01.488 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T17:07:01.488 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 672 M 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 3.1 G 2024-03-25T17:07:01.489 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T17:08:21.921 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 254 kB/s | 176 MB 11:48 2024-03-25T17:08:21.922 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:21.922 INFO:teuthology.orchestra.run.smithi161.stdout:Total 254 kB/s | 176 MB 11:48 2024-03-25T17:08:21.922 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T17:08:21.926 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T17:08:21.926 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T17:08:22.983 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T17:08:22.983 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T17:08:40.935 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T17:08:41.099 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:45.498 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:45.759 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:45.759 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:08:45.759 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T17:08:45.759 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:45.759 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:08:45.760 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T17:08:45.973 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:08:46.538 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 0:45:30 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T17:08:46.688 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 672 M 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 3.1 G 2024-03-25T17:08:46.689 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T17:36:10.680 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 419 kB/s | 672 MB 27:23 2024-03-25T17:36:10.703 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:10.703 INFO:teuthology.orchestra.run.smithi161.stdout:Total 419 kB/s | 672 MB 27:23 2024-03-25T17:36:10.704 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T17:36:10.704 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T17:36:10.704 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T17:36:14.652 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T17:36:14.653 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T17:37:25.494 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T17:37:25.845 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:35.505 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:35.763 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:35.763 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:37:35.763 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T17:37:35.763 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:35.763 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:37:35.763 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T17:37:36.032 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:37:36.599 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:14:20 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T17:37:36.697 INFO:teuthology.orchestra.run.smithi161.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:37:36.748 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T17:37:36.749 INFO:teuthology.orchestra.run.smithi161.stdout:Nothing to do. 2024-03-25T17:37:36.749 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T17:37:36.809 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:37:37.378 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:14:21 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T17:37:37.528 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T17:37:37.528 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:37:37.528 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 20 M 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 103 M 2024-03-25T17:37:37.529 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T17:38:28.093 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 399 kB/s | 20 MB 00:50 2024-03-25T17:38:28.094 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:28.094 INFO:teuthology.orchestra.run.smithi161.stdout:Total 399 kB/s | 20 MB 00:50 2024-03-25T17:38:28.095 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T17:38:28.097 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T17:38:28.097 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T17:38:28.246 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T17:38:28.246 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T17:38:30.467 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T17:38:30.656 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:31.562 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:31.828 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:31.828 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:38:31.828 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T17:38:31.829 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:31.829 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:38:31.829 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T17:38:31.995 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:38:32.567 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:15:16 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T17:38:32.715 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T17:38:32.716 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:38:32.717 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 48 M 2024-03-25T17:38:32.717 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 173 M 2024-03-25T17:38:32.717 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T17:40:57.876 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 337 kB/s | 48 MB 02:25 2024-03-25T17:40:57.877 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:57.877 INFO:teuthology.orchestra.run.smithi161.stdout:Total 337 kB/s | 48 MB 02:25 2024-03-25T17:40:57.877 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T17:40:57.880 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T17:40:57.880 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T17:40:58.195 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T17:40:58.195 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T17:41:02.948 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T17:41:03.056 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:04.334 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:04.602 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:41:04.602 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:41:04.603 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T17:41:04.603 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:04.603 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:41:04.603 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T17:41:04.771 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:41:05.343 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:17:49 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T17:41:05.491 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T17:41:05.492 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:41:05.492 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T17:41:05.492 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 99 M 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 449 M 2024-03-25T17:41:05.493 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T17:41:15.239 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 335 kB/s | 672 MB 34:13 2024-03-25T17:41:15.239 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:15.240 INFO:teuthology.orchestra.run.smithi132.stdout:Total 335 kB/s | 672 MB 34:13 2024-03-25T17:41:15.240 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T17:41:15.245 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T17:41:15.245 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T17:41:19.196 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T17:41:19.196 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T17:42:29.835 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T17:42:29.948 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:42:38.983 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:42:39.280 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:42:39.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:42:39.280 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T17:42:39.280 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:42:39.280 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:42:39.281 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T17:42:39.569 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:42:40.152 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:19:48 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T17:42:40.256 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:42:40.313 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T17:42:40.314 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2024-03-25T17:42:40.314 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T17:42:40.437 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:42:41.027 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:19:49 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T17:42:41.183 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T17:42:41.184 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:42:41.184 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T17:42:41.184 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 20 M 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 103 M 2024-03-25T17:42:41.185 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T17:43:38.945 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 349 kB/s | 20 MB 00:57 2024-03-25T17:43:38.946 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:38.946 INFO:teuthology.orchestra.run.smithi132.stdout:Total 349 kB/s | 20 MB 00:57 2024-03-25T17:43:38.946 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T17:43:38.950 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T17:43:38.950 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T17:43:39.100 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T17:43:39.101 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T17:43:41.321 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T17:43:41.460 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:42.489 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:42.751 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:42.752 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:43:42.752 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T17:43:42.752 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:42.752 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:43:42.752 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T17:43:42.928 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:43:43.493 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:20:51 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T17:43:43.641 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T17:43:43.641 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 48 M 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 173 M 2024-03-25T17:43:43.642 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T17:45:26.751 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 386 kB/s | 99 MB 04:21 2024-03-25T17:45:26.751 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:26.751 INFO:teuthology.orchestra.run.smithi161.stdout:Total 386 kB/s | 99 MB 04:21 2024-03-25T17:45:26.751 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T17:45:26.754 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T17:45:26.754 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T17:45:27.362 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T17:45:27.363 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T17:45:37.629 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T17:45:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:38.615 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 425 kB/s | 48 MB 01:54 2024-03-25T17:45:38.616 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:45:38.616 INFO:teuthology.orchestra.run.smithi132.stdout:Total 425 kB/s | 48 MB 01:54 2024-03-25T17:45:38.616 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T17:45:38.620 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T17:45:38.620 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T17:45:38.934 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T17:45:38.934 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T17:45:40.507 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:45:40.782 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T17:45:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:40.782 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:45:40.782 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T17:45:40.944 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:45:41.510 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:22:25 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T17:45:41.657 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T17:45:41.658 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:45:41.658 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T17:45:41.658 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:45:41.658 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T17:45:41.658 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 287 M 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.2 G 2024-03-25T17:45:41.659 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T17:45:43.650 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T17:45:43.813 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:45.234 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:45.500 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:45.500 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:45:45.501 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T17:45:45.501 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:45.501 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:45:45.501 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T17:45:45.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:45:46.258 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:22:54 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T17:45:46.405 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T17:45:46.406 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 99 M 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 449 M 2024-03-25T17:45:46.407 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T17:49:35.730 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 440 kB/s | 99 MB 03:49 2024-03-25T17:49:35.730 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:35.730 INFO:teuthology.orchestra.run.smithi132.stdout:Total 440 kB/s | 99 MB 03:49 2024-03-25T17:49:35.730 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T17:49:35.733 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T17:49:35.733 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T17:49:36.342 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T17:49:36.342 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T17:49:46.598 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T17:49:46.692 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:49.737 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:50.036 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:50.036 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:49:50.036 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T17:49:50.036 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:50.037 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:49:50.037 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T17:49:50.226 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:49:50.795 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:26:58 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T17:49:50.944 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T17:49:50.945 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T17:49:50.946 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 287 M 2024-03-25T17:49:50.946 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.2 G 2024-03-25T17:49:50.946 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T17:59:44.678 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 495 kB/s | 287 MB 09:53 2024-03-25T17:59:44.679 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:44.679 INFO:teuthology.orchestra.run.smithi132.stdout:Total 495 kB/s | 287 MB 09:53 2024-03-25T17:59:44.679 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T17:59:44.682 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T17:59:44.682 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T17:59:46.393 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T17:59:46.394 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:00:15.736 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:00:16.350 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:21.622 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:21.901 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:00:21.901 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:21.902 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:00:21.902 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:21.902 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:21.902 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:00:22.129 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:00:22.694 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:37:30 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:00:22.841 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:00:22.841 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 5.9 M 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 28 M 2024-03-25T18:00:22.842 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:00:35.109 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 496 kB/s | 5.9 MB 00:12 2024-03-25T18:00:35.110 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:35.110 INFO:teuthology.orchestra.run.smithi132.stdout:Total 496 kB/s | 5.9 MB 00:12 2024-03-25T18:00:35.110 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:00:35.114 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:00:35.114 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:00:35.185 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:00:35.186 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:00:35.907 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:00:36.024 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:36.639 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:36.900 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:36.900 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:36.900 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:00:36.900 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:36.900 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:36.900 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:00:37.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:00:37.677 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:37:45 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:00:37.826 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 109 k 2024-03-25T18:00:37.827 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 440 k 2024-03-25T18:00:37.828 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:00:38.870 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 104 kB/s | 109 kB 00:01 2024-03-25T18:00:38.871 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:38.871 INFO:teuthology.orchestra.run.smithi132.stdout:Total 104 kB/s | 109 kB 00:01 2024-03-25T18:00:38.871 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:00:38.875 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:00:38.875 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:00:38.911 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:00:38.912 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:00:39.027 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:00:39.135 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:00:39.528 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:00:39.791 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:00:39.791 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:39.791 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:00:39.791 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:39.791 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:39.792 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:00:39.975 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:00:40.543 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:37:48 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:00:40.693 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:00:40.693 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 5.4 M 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 25 M 2024-03-25T18:00:40.694 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:00:52.030 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 483 kB/s | 5.4 MB 00:11 2024-03-25T18:00:52.031 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:00:52.031 INFO:teuthology.orchestra.run.smithi132.stdout:Total 483 kB/s | 5.4 MB 00:11 2024-03-25T18:00:52.032 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:00:52.036 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:00:52.036 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:00:52.105 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:00:52.105 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:00:52.770 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:00:52.887 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:53.468 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:53.732 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:00:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:53.733 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:00:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:53.733 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:53.733 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:00:53.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2-debuginfo 2024-03-25T18:00:54.495 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:38:02 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:00:54.644 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 44 M 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 253 M 2024-03-25T18:00:54.645 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:02:54.012 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 284 kB/s | 287 MB 17:12 2024-03-25T18:02:54.013 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:02:54.013 INFO:teuthology.orchestra.run.smithi161.stdout:Total 284 kB/s | 287 MB 17:12 2024-03-25T18:02:54.013 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:02:54.016 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:02:54.016 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:02:55.720 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:02:55.720 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:03:17.222 INFO:teuthology.orchestra.run.smithi132.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 318 kB/s | 44 MB 02:22 2024-03-25T18:03:17.223 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:17.223 INFO:teuthology.orchestra.run.smithi132.stdout:Total 318 kB/s | 44 MB 02:22 2024-03-25T18:03:17.223 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:03:17.227 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:03:17.227 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:03:17.525 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:03:17.525 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:03:22.578 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:03:22.670 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:03:24.566 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:03:24.832 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:03:24.832 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:03:24.832 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:03:24.832 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:24.832 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:03:24.832 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:03:25.060 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:03:25.150 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:03:25.457 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:25.620 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:40:33 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:03:25.770 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:03:25.771 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:03:25.772 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:03:25.772 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 52 M 2024-03-25T18:03:25.772 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 308 M 2024-03-25T18:03:25.772 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:03:32.678 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:32.969 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:32.969 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:32.969 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:03:32.969 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:32.969 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:32.969 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:03:33.168 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:03:33.737 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:40:17 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:03:33.886 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:03:33.886 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:33.886 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:03:33.886 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:33.886 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 5.9 M 2024-03-25T18:03:33.887 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 28 M 2024-03-25T18:03:33.888 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:03:54.291 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 298 kB/s | 5.9 MB 00:20 2024-03-25T18:03:54.292 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:54.292 INFO:teuthology.orchestra.run.smithi161.stdout:Total 298 kB/s | 5.9 MB 00:20 2024-03-25T18:03:54.292 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:03:54.296 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:03:54.297 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:03:54.368 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:03:54.368 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:03:55.093 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:03:55.188 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:03:55.795 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:03:56.073 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:03:56.073 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:56.073 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:03:56.073 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:56.073 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:56.074 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:03:56.255 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:03:56.822 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:40:40 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:03:56.971 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 109 k 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 440 k 2024-03-25T18:03:56.972 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:03:57.919 INFO:teuthology.orchestra.run.smithi161.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 115 kB/s | 109 kB 00:00 2024-03-25T18:03:57.919 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:57.919 INFO:teuthology.orchestra.run.smithi161.stdout:Total 115 kB/s | 109 kB 00:00 2024-03-25T18:03:57.920 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:03:57.924 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:03:57.924 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:03:57.961 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:03:57.961 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:03:58.085 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:03:58.192 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:58.560 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:58.839 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:58.839 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:58.839 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:03:58.839 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:58.839 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:58.840 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:03:59.040 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:03:59.603 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:40:43 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:03:59.752 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:03:59.753 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:59.753 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 5.4 M 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 25 M 2024-03-25T18:03:59.754 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:04:15.795 INFO:teuthology.orchestra.run.smithi161.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 342 kB/s | 5.4 MB 00:16 2024-03-25T18:04:15.796 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:15.796 INFO:teuthology.orchestra.run.smithi161.stdout:Total 342 kB/s | 5.4 MB 00:16 2024-03-25T18:04:15.796 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:04:15.801 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:04:15.801 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:04:15.870 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:04:15.871 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:04:16.551 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:04:16.652 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:17.172 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:17.463 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:17.463 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:04:17.463 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:04:17.463 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:17.463 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:04:17.463 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:04:17.674 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librados2-debuginfo 2024-03-25T18:04:18.241 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:41:02 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:04:18.389 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:04:18.390 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:04:18.391 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:04:18.392 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:04:18.392 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 44 M 2024-03-25T18:04:18.392 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 253 M 2024-03-25T18:04:18.392 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:05:31.968 INFO:teuthology.orchestra.run.smithi132.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 421 kB/s | 52 MB 02:06 2024-03-25T18:05:31.968 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:31.968 INFO:teuthology.orchestra.run.smithi132.stdout:Total 421 kB/s | 52 MB 02:06 2024-03-25T18:05:31.968 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:05:31.973 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:05:31.973 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:05:32.314 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:05:32.314 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:05:38.278 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:05:38.506 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:40.738 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:41.004 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:05:41.004 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:41.004 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:05:41.004 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:41.004 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:41.004 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:05:41.223 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:05:41.779 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:42:49 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:05:41.928 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:05:41.929 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:41.929 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:05:41.929 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 615 k 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.1 M 2024-03-25T18:05:41.930 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:05:43.427 INFO:teuthology.orchestra.run.smithi132.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 411 kB/s | 615 kB 00:01 2024-03-25T18:05:43.428 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:43.428 INFO:teuthology.orchestra.run.smithi132.stdout:Total 410 kB/s | 615 kB 00:01 2024-03-25T18:05:43.429 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:05:43.433 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:05:43.433 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:05:43.473 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:05:43.474 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:05:43.636 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:05:43.742 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:44.170 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:44.427 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:05:44.427 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:44.428 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:44.428 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:05:44.624 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:05:45.188 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:42:53 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:05:45.338 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:45.339 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:05:45.340 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:45.340 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 771 k 2024-03-25T18:05:45.340 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.7 M 2024-03-25T18:05:45.340 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:05:48.855 INFO:teuthology.orchestra.run.smithi132.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 219 kB/s | 771 kB 00:03 2024-03-25T18:05:48.856 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:48.856 INFO:teuthology.orchestra.run.smithi132.stdout:Total 219 kB/s | 771 kB 00:03 2024-03-25T18:05:48.856 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:05:48.860 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:05:48.860 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:05:48.901 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:05:48.901 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:05:49.085 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:05:49.196 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:49.561 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:49.827 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:49.828 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:49.828 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:05:49.828 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:49.828 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:49.828 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:05:50.044 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:05:50.606 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:42:58 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:05:50.756 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:05:50.756 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:50.756 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:05:50.756 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:50.756 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:05:50.756 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.1 M 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 4.3 M 2024-03-25T18:05:50.757 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:05:57.416 INFO:teuthology.orchestra.run.smithi132.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 173 kB/s | 1.1 MB 00:06 2024-03-25T18:05:57.417 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:57.417 INFO:teuthology.orchestra.run.smithi132.stdout:Total 173 kB/s | 1.1 MB 00:06 2024-03-25T18:05:57.417 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:05:57.421 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:05:57.421 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:05:57.464 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:05:57.464 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:05:57.688 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:05:57.806 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:58.234 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:58.492 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:58.492 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:58.492 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:05:58.492 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:58.492 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:58.492 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:05:58.746 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:05:59.308 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:43:07 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:05:59.459 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 337 k 2024-03-25T18:05:59.460 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 1.0 M 2024-03-25T18:05:59.461 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:06:01.394 INFO:teuthology.orchestra.run.smithi132.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 175 kB/s | 337 kB 00:01 2024-03-25T18:06:01.395 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:01.395 INFO:teuthology.orchestra.run.smithi132.stdout:Total 174 kB/s | 337 kB 00:01 2024-03-25T18:06:01.395 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:06:01.399 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:06:01.399 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:06:01.438 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:06:01.438 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:06:01.575 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:06:01.701 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:06:02.059 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:06:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:06:02.316 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:02.316 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:06:02.317 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:02.317 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:02.317 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:06:02.549 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:06:03.109 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:43:11 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:06:03.258 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:06:03.259 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:03.260 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 509 k 2024-03-25T18:06:03.260 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 2.5 M 2024-03-25T18:06:03.260 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:06:05.680 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 210 kB/s | 509 kB 00:02 2024-03-25T18:06:05.680 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:05.681 INFO:teuthology.orchestra.run.smithi132.stdout:Total 210 kB/s | 509 kB 00:02 2024-03-25T18:06:05.681 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:06:05.685 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:06:05.685 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:06:05.724 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:06:05.724 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:06:05.889 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:06:05.986 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:06.392 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:06.648 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:06.648 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:06.648 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:06:06.648 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:06.648 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:06.648 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:06:06.891 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:06:07.455 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:43:15 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:06:07.605 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:06:07.606 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:06:07.607 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:06:07.607 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 52 M 2024-03-25T18:06:07.607 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 298 M 2024-03-25T18:06:07.607 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:06:49.185 INFO:teuthology.orchestra.run.smithi161.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 301 kB/s | 44 MB 02:30 2024-03-25T18:06:49.186 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:49.187 INFO:teuthology.orchestra.run.smithi161.stdout:Total 301 kB/s | 44 MB 02:30 2024-03-25T18:06:49.187 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:06:49.191 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:06:49.191 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:06:49.491 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:06:49.491 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:06:54.546 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:06:54.643 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:56.465 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:06:56.746 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:06:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:56.746 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:06:56.746 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:06:56.942 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:06:57.506 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:43:41 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:06:57.655 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:06:57.656 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:06:57.657 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:06:57.657 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:06:57.657 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:06:57.657 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 52 M 2024-03-25T18:06:57.657 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 308 M 2024-03-25T18:06:57.657 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:08:42.197 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 341 kB/s | 52 MB 02:34 2024-03-25T18:08:42.197 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:42.197 INFO:teuthology.orchestra.run.smithi132.stdout:Total 341 kB/s | 52 MB 02:34 2024-03-25T18:08:42.198 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:08:42.200 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:08:42.200 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:08:42.534 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:08:42.535 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:08:48.432 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:08:48.526 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:08:50.622 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:08:50.897 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:08:50.898 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:08:50.898 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:08:50.898 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:50.898 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:08:50.898 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:08:51.114 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:08:51.682 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 1:45:59 ago on Mon 25 Mar 2024 04:22:52 PM UTC. 2024-03-25T18:08:51.830 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2024-03-25T18:08:51.830 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:08:51.830 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2024-03-25T18:08:51.831 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:08:51.832 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.3 M 2024-03-25T18:08:51.832 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 5.5 M 2024-03-25T18:08:51.832 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2024-03-25T18:08:55.631 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 337 kB/s | 1.3 MB 00:03 2024-03-25T18:08:55.631 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:08:55.632 INFO:teuthology.orchestra.run.smithi132.stdout:Total 337 kB/s | 1.3 MB 00:03 2024-03-25T18:08:55.632 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2024-03-25T18:08:55.636 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2024-03-25T18:08:55.636 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2024-03-25T18:08:55.679 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2024-03-25T18:08:55.680 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2024-03-25T18:08:55.936 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2024-03-25T18:08:56.054 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:56.473 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:56.730 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:08:56.730 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:08:56.731 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2024-03-25T18:08:56.731 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:08:56.731 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:08:56.731 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2024-03-25T18:08:56.943 DEBUG:teuthology.parallel:result is None 2024-03-25T18:09:49.742 INFO:teuthology.orchestra.run.smithi161.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 308 kB/s | 52 MB 02:52 2024-03-25T18:09:49.743 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:49.743 INFO:teuthology.orchestra.run.smithi161.stdout:Total 308 kB/s | 52 MB 02:52 2024-03-25T18:09:49.743 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:09:49.747 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:09:49.747 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:09:50.089 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:09:50.090 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:09:56.080 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:09:56.432 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:58.609 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:58.874 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:58.874 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:09:58.874 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:09:58.874 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:58.874 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:09:58.874 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:09:59.073 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:09:59.632 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:46:43 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:09:59.784 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:09:59.785 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:09:59.786 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 615 k 2024-03-25T18:09:59.786 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 2.1 M 2024-03-25T18:09:59.786 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:10:01.678 INFO:teuthology.orchestra.run.smithi161.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 325 kB/s | 615 kB 00:01 2024-03-25T18:10:01.678 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:01.679 INFO:teuthology.orchestra.run.smithi161.stdout:Total 325 kB/s | 615 kB 00:01 2024-03-25T18:10:01.679 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:10:01.683 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:10:01.683 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:10:01.725 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:10:01.726 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:10:01.920 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:10:02.036 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:02.396 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:02.665 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:10:02.665 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:02.665 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:10:02.665 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:02.665 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:02.666 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:10:02.849 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:10:03.414 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:46:47 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:10:03.567 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:10:03.567 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:03.567 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:10:03.567 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:03.567 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:10:03.567 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 771 k 2024-03-25T18:10:03.568 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 2.7 M 2024-03-25T18:10:03.569 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:10:07.688 INFO:teuthology.orchestra.run.smithi161.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 187 kB/s | 771 kB 00:04 2024-03-25T18:10:07.689 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:07.689 INFO:teuthology.orchestra.run.smithi161.stdout:Total 187 kB/s | 771 kB 00:04 2024-03-25T18:10:07.690 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:10:07.693 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:10:07.693 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:10:07.734 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:10:07.735 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:10:07.917 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:10:08.029 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:08.403 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:08.714 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:10:08.714 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:08.714 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:10:08.714 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:08.714 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:08.714 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:10:08.926 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:10:09.491 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:46:53 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:10:09.642 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:10:09.642 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:10:09.643 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:09.644 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 1.1 M 2024-03-25T18:10:09.644 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 4.3 M 2024-03-25T18:10:09.644 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:10:12.458 INFO:teuthology.orchestra.run.smithi161.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 409 kB/s | 1.1 MB 00:02 2024-03-25T18:10:12.458 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:12.459 INFO:teuthology.orchestra.run.smithi161.stdout:Total 409 kB/s | 1.1 MB 00:02 2024-03-25T18:10:12.459 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:10:12.463 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:10:12.463 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:10:12.507 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:10:12.508 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:10:12.728 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:10:12.832 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:13.201 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:13.473 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:13.473 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:13.473 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:10:13.473 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:13.473 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:13.473 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:10:13.683 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:10:14.249 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:46:58 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:10:14.401 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:10:14.402 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:14.402 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:10:14.402 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:14.402 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:10:14.402 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 337 k 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 1.0 M 2024-03-25T18:10:14.403 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:10:15.869 INFO:teuthology.orchestra.run.smithi161.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 230 kB/s | 337 kB 00:01 2024-03-25T18:10:15.869 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:15.870 INFO:teuthology.orchestra.run.smithi161.stdout:Total 230 kB/s | 337 kB 00:01 2024-03-25T18:10:15.870 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:10:15.874 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:10:15.874 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:10:15.914 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:10:15.915 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:10:16.058 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:10:16.168 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:16.517 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:16.780 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:16.780 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:16.780 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:10:16.780 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:16.780 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:16.780 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:10:16.983 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:10:17.544 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:47:01 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:10:17.691 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:10:17.692 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:17.693 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 509 k 2024-03-25T18:10:17.693 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 2.5 M 2024-03-25T18:10:17.693 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:10:19.546 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 275 kB/s | 509 kB 00:01 2024-03-25T18:10:19.547 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:19.547 INFO:teuthology.orchestra.run.smithi161.stdout:Total 274 kB/s | 509 kB 00:01 2024-03-25T18:10:19.548 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:10:19.552 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:10:19.552 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:10:19.592 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:10:19.592 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:10:19.755 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:10:19.854 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:20.238 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:20.504 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:20.504 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:20.504 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:10:20.504 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:20.504 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:20.504 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:10:20.685 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:10:21.244 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:47:05 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:21.393 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 52 M 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 298 M 2024-03-25T18:10:21.394 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:12:22.901 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 434 kB/s | 52 MB 02:01 2024-03-25T18:12:22.901 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:22.901 INFO:teuthology.orchestra.run.smithi161.stdout:Total 434 kB/s | 52 MB 02:01 2024-03-25T18:12:22.901 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:12:22.904 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:12:22.904 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:12:23.238 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:12:23.238 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:12:29.142 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:12:29.381 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:12:31.377 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:12:31.650 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:12:31.650 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:31.650 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:12:31.651 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:31.651 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:31.651 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:12:31.811 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:12:32.386 INFO:teuthology.orchestra.run.smithi161.stdout:Last metadata expiration check: 1:49:16 ago on Mon 25 Mar 2024 04:23:16 PM UTC. 2024-03-25T18:12:32.544 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repo Size 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:Install 1 Package 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 1.3 M 2024-03-25T18:12:32.545 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 5.5 M 2024-03-25T18:12:32.546 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-25T18:12:37.392 INFO:teuthology.orchestra.run.smithi161.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 265 kB/s | 1.3 MB 00:04 2024-03-25T18:12:37.393 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:37.393 INFO:teuthology.orchestra.run.smithi161.stdout:Total 264 kB/s | 1.3 MB 00:04 2024-03-25T18:12:37.394 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-25T18:12:37.398 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-25T18:12:37.398 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-25T18:12:37.442 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-25T18:12:37.443 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-25T18:12:37.692 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-25T18:12:37.782 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:12:38.191 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:12:38.458 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:12:38.459 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:38.459 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-25T18:12:38.459 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:38.459 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:38.459 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-25T18:12:38.641 DEBUG:teuthology.parallel:result is None 2024-03-25T18:12:38.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:12:38.868 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:12:38.893 INFO:teuthology.orchestra.run.smithi132.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:12:38.893 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:12:38.893 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:12:38.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:12:39.034 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:12:39.055 INFO:teuthology.orchestra.run.smithi161.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:12:39.055 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:12:39.055 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:12:39.057 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:12:39.057 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:39.057 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:12:39.088 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:39.088 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:12:39.123 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:12:39.123 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:39.123 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:12:39.158 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:12:39.228 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:39.228 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:12:39.255 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:12:39.322 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:12:39.322 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:39.322 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:12:39.352 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:12:39.420 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:39.420 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:12:39.446 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:12:39.513 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:12:39.513 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:39.513 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:12:39.539 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:12:39.606 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:39.606 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:12:39.632 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:12:39.700 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:12:39.779 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:12:39.779 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /var/log/ceph 2024-03-25T18:12:39.781 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2024-03-25T18:12:39.807 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:12:39.808 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:12:39.849 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:12:39.875 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:12:39.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:12:39.915 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:12:39.983 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:12:39.983 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:12:39.983 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621260', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi132.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDyGPtVmN/t/hScr/Lh41lhd2ESJ/noBK6bWwPWjF49cnIjlANMDM1i0OWQJ541/5QR58IysledB17AshalkgKs=', 'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5lvloWX4M/z9MMgX4Co9KLPl1cnAOYTbRK0YDa1V8wVIVlmbo+CfY4RPrgGwOTj9EaNdDBbQRWlsZTtY4Fd28='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:12:39.983 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:12:39.986 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:12:40.000 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:12:40.041 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:12:40.067 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:40.067 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:12:40.120 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:12:40.121 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_1 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-25 16:27:46.612274283 +0000 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-25 16:22:03.710943857 +0000 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-25 16:22:03.710943857 +0000 2024-03-25T18:12:40.175 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-25 16:22:03.710943857 +0000 2024-03-25T18:12:40.176 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:12:40.240 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-25T18:12:40.240 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-25T18:12:40.241 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.00026572 s, 1.9 MB/s 2024-03-25T18:12:40.242 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:12:40.296 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_2 2024-03-25T18:12:40.350 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-25 16:27:46.612274283 +0000 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-25 16:22:04.063934374 +0000 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-25 16:22:04.063934374 +0000 2024-03-25T18:12:40.351 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-25 16:22:04.063934374 +0000 2024-03-25T18:12:40.351 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:12:40.415 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-25T18:12:40.415 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-25T18:12:40.416 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000186962 s, 2.7 MB/s 2024-03-25T18:12:40.417 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:12:40.472 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_3 2024-03-25T18:12:40.526 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-25 16:27:46.613274257 +0000 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-25 16:22:04.397925402 +0000 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-25 16:22:04.397925402 +0000 2024-03-25T18:12:40.527 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-25 16:22:04.397925402 +0000 2024-03-25T18:12:40.527 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:12:40.593 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-25T18:12:40.593 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-25T18:12:40.593 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.00019466 s, 2.6 MB/s 2024-03-25T18:12:40.594 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:12:40.650 DEBUG:teuthology.orchestra.run.smithi132:> stat /dev/vg_nvme/lv_4 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout:Access: 2024-03-25 16:27:46.613274257 +0000 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout:Modify: 2024-03-25 16:22:04.752915862 +0000 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout:Change: 2024-03-25 16:22:04.752915862 +0000 2024-03-25T18:12:40.705 INFO:teuthology.orchestra.run.smithi132.stdout: Birth: 2024-03-25 16:22:04.752915862 +0000 2024-03-25T18:12:40.705 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:12:40.770 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records in 2024-03-25T18:12:40.771 INFO:teuthology.orchestra.run.smithi132.stderr:1+0 records out 2024-03-25T18:12:40.771 INFO:teuthology.orchestra.run.smithi132.stderr:512 bytes copied, 0.000183608 s, 2.8 MB/s 2024-03-25T18:12:40.772 DEBUG:teuthology.orchestra.run.smithi132:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:12:40.828 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:40.828 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:40.828 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:12:40.844 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:12:40.844 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-03-25T18:12:40.899 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:12:40.899 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:40.899 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:12:40.900 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:40.900 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:40.900 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-25 16:28:42.865999994 +0000 2024-03-25T18:12:40.900 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-25 16:21:58.492214271 +0000 2024-03-25T18:12:40.900 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-25 16:21:58.492214271 +0000 2024-03-25T18:12:40.900 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-25 16:21:58.492214271 +0000 2024-03-25T18:12:40.900 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:12:40.966 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-25T18:12:40.966 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-25T18:12:40.966 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000217036 s, 2.4 MB/s 2024-03-25T18:12:40.967 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:12:41.023 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-03-25T18:12:41.078 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:12:41.078 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:41.078 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:12:41.078 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:41.078 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:41.079 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-25 16:28:42.865999994 +0000 2024-03-25T18:12:41.079 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-25 16:21:58.824207161 +0000 2024-03-25T18:12:41.079 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-25 16:21:58.824207161 +0000 2024-03-25T18:12:41.079 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-25 16:21:58.824207161 +0000 2024-03-25T18:12:41.079 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:12:41.144 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-25T18:12:41.144 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-25T18:12:41.144 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000201139 s, 2.5 MB/s 2024-03-25T18:12:41.145 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:12:41.200 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-25 16:28:42.866999973 +0000 2024-03-25T18:12:41.255 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-25 16:21:59.169199772 +0000 2024-03-25T18:12:41.256 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-25 16:21:59.169199772 +0000 2024-03-25T18:12:41.256 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-25 16:21:59.169199772 +0000 2024-03-25T18:12:41.256 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:12:41.321 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-25T18:12:41.321 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-25T18:12:41.321 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000184058 s, 2.8 MB/s 2024-03-25T18:12:41.322 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:12:41.377 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-25 16:28:42.866999973 +0000 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-25 16:21:59.697188463 +0000 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-25 16:21:59.697188463 +0000 2024-03-25T18:12:41.432 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-25 16:21:59.697188463 +0000 2024-03-25T18:12:41.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:12:41.497 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-25T18:12:41.497 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-25T18:12:41.497 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000192947 s, 2.7 MB/s 2024-03-25T18:12:41.499 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:12:41.554 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:41.554 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:12:41.555 INFO:tasks.ceph:Generating config... 2024-03-25T18:12:41.556 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:12:41.556 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:12:41.556 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:12:41.556 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:12:41.556 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:12:41.556 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:12:41.556 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:12:41.556 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:12:41.556 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:12:41.556 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:12:41.556 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:12:41.556 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:12:41.556 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:12:41.556 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:12:41.557 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:12:41.557 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:12:41.558 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:12:41.559 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:12:41.559 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:12:41.559 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:12:41.616 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:12:41.619 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:12:41.664 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:12:41.691 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.132:6789]'), ('mon.c', '[v1:172.21.15.132:6790]'), ('mon.b', '[v1:172.21.15.161:6789]')] 2024-03-25T18:12:41.692 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.132:6789],[v1:172.21.15.132:6790],[v1:172.21.15.161:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:12:41.692 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:41.693 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:12:41.746 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.132:6789]' --addv c '[v1:172.21.15.132:6790]' --addv b '[v1:172.21.15.161:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:41.830 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:41.830 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: generated fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:12:41.830 INFO:teuthology.orchestra.run.smithi132.stdout:setting min_mon_release = quincy 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:epoch 0 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:min_mon_release 17 (quincy) 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:election_strategy: 3 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:12:41.831 INFO:teuthology.orchestra.run.smithi132.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:12:41.832 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:12:41.847 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 6e953815-4f55-4f47-8818-b2f2072285a9... 2024-03-25T18:12:41.847 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:12:41.889 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:12:41.936 INFO:teuthology.orchestra.run.smithi132.stdout:[global] 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: chdir = "" 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: auth supported = cephx 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: filestore xattr use omap = true 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: mon clock drift allowed = 1.000 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: osd crush chooseleaf type = 0 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: auth debug = true 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on old message = true 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: ms die on bug = true 2024-03-25T18:12:41.937 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon pg warn max object skew = 0 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default size = 2 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow primary affinity = true 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd allow pg remap = true 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on no sortbitwise = false 2024-03-25T18:12:41.938 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on too few osds = false 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon_allow_pool_size_one = true 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: osd default data pool replay window = 5 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon allow pool delete = true 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon cluster log file level = debug 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: debug asserts on shutdown = true 2024-03-25T18:12:41.939 INFO:teuthology.orchestra.run.smithi132.stdout: mon health detail to clog = false 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: mon host = "[v1:172.21.15.132:6789],[v1:172.21.15.132:6790],[v1:172.21.15.161:6789]" 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: mon client directed command retry = 5 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: mon election default strategy = 3 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: ms bind msgr2 = False 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: ms inject socket failures = 5000 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: ms type = async 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: osd heartbeat grace = 80 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: fsid = 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout:[osd] 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: osd journal size = 100 2024-03-25T18:12:41.940 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub load threshold = 5.0 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd scrub max interval = 600 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd recover clone overlap = true 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd recovery max chunk = 1048576 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug shutdown = true 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug op order = true 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify stray on activate = true 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug trim objects = true 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd open classes on start = true 2024-03-25T18:12:41.941 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug pg log writeout = true 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: osd map max advance = 10 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: journal zero on create = true 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: filestore apply finisher threads = 3 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: bdev debug aio = true 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug misdirected ops = true 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: bdev async discard = True 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: bdev enable discard = True 2024-03-25T18:12:41.942 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore allocator = stupid 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore block size = 96636764160 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore fsck on mount = True 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: bluestore zero block detection = True 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluefs = 1 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: debug bluestore = 1 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: debug monc = 20 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: debug osd = 20 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: debug rocksdb = 10 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd full ratio = 0.9 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: osd class default list = * 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: osd class load list = * 2024-03-25T18:12:41.943 INFO:teuthology.orchestra.run.smithi132.stdout: osd client watch timeout = 120 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify cached snaps = True 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd debug verify missing on start = True 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd fast shutdown = False 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd max markdown count = 1000 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd mclock override recovery settings = True 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd objectstore = bluestore 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd op queue = debug_random 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd op queue cut off = debug_random 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: osd shutdown pgref assert = True 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr] 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-25T18:12:41.944 INFO:teuthology.orchestra.run.smithi132.stdout: debug mgr = 20 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: mgr/telemetry/nag = false 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout:[mon] 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: debug mon = 20 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: debug paxos = 20 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: debug auth = 20 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: mon data avail warn = 5 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: mon mgr mkfs grace = 240 2024-03-25T18:12:41.945 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd reporter subtree level = osd 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd prime pg temp = true 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: # 1m isn't quite enough 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:12:41.946 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon min osdmap epochs = 50 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon osd crush smoke test = False 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon osdmap full prune interval = 2 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon osdmap full prune min = 15 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon scrub interval = 300 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: mon warn on pool no app = False 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: paxos service trim min = 10 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout:[client] 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: rgw cache enabled = true 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable ops log = true 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: rgw enable usage log = true 2024-03-25T18:12:41.947 INFO:teuthology.orchestra.run.smithi132.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout: debug monc = 20 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout: debug ms = 1 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout: debug objecter = 20 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout: debug rados = 20 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.a] 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.c] 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi132.stdout:[mon.b] 2024-03-25T18:12:41.948 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2024-03-25T18:12:41.949 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:12:41.950 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2024-03-25T18:12:41.951 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = "[v1:172.21.15.132:6789],[v1:172.21.15.132:6790],[v1:172.21.15.161:6789]" 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: mon client directed command retry = 5 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: mon election default strategy = 3 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: ms bind msgr2 = False 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: ms inject socket failures = 5000 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: ms type = async 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: osd heartbeat grace = 80 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2024-03-25T18:12:41.952 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug trim objects = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:12:41.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: bdev async discard = True 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: bdev enable discard = True 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore allocator = stupid 2024-03-25T18:12:41.954 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore block size = 96636764160 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore fsck on mount = True 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore zero block detection = True 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 1 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 1 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: debug monc = 20 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 10 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd full ratio = 0.9 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: osd class default list = * 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: osd class load list = * 2024-03-25T18:12:41.955 INFO:teuthology.orchestra.run.smithi161.stdout: osd client watch timeout = 120 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify cached snaps = True 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify missing on start = True 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd fast shutdown = False 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd max markdown count = 1000 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock override recovery settings = True 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd objectstore = bluestore 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd op queue = debug_random 2024-03-25T18:12:41.956 INFO:teuthology.orchestra.run.smithi161.stdout: osd op queue cut off = debug_random 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: osd shutdown pgref assert = True 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-03-25T18:12:41.957 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:12:41.958 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon min osdmap epochs = 50 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd crush smoke test = False 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon osdmap full prune interval = 2 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon osdmap full prune min = 15 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon scrub interval = 300 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on pool no app = False 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: paxos service trim min = 10 2024-03-25T18:12:41.959 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: debug monc = 20 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: debug objecter = 20 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout: debug rados = 20 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.c] 2024-03-25T18:12:41.960 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.b] 2024-03-25T18:12:41.962 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:12:41.962 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:12:42.016 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:12:42.016 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:42.017 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:12:42.031 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:42.031 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:12:42.085 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:12:42.085 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:42.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:12:42.086 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:12:42.164 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:12:42.164 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:42.219 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:12:42.219 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:42.219 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:12:42.219 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:12:42.259 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:12:42.259 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:42.314 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:12:42.314 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:12:42.314 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:12:42.368 INFO:teuthology.orchestra.run.smithi132.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:12:42.371 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:12:42.435 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:12:42.437 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:12:42.437 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:12:42.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:12:42.479 INFO:teuthology.orchestra.run.smithi132.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:12:42.495 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:12:42.537 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:12:42.552 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:12:42.552 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi132.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:12:42.552 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:12:42.578 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:42.578 INFO:tasks.ceph:role: osd.0 2024-03-25T18:12:42.578 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:12:42.579 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:12:42.652 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:42.652 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:42.652 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:42.652 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:42.652 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:42.652 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:42.653 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:42.653 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:42.653 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:42.653 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:43.139 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-25T18:12:43.141 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-25T18:12:43.141 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:12:43.288 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:12:43.323 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:12:43.393 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:43.393 INFO:tasks.ceph:role: osd.1 2024-03-25T18:12:43.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:12:43.394 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:43.470 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:43.929 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-25T18:12:43.931 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-25T18:12:43.931 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:12:44.025 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:12:44.058 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:12:44.127 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:44.127 INFO:tasks.ceph:role: osd.2 2024-03-25T18:12:44.127 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:12:44.128 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:12:44.202 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:44.202 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:44.203 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:44.694 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-25T18:12:44.697 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-25T18:12:44.697 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:12:44.840 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:12:44.875 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:12:44.943 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:44.943 INFO:tasks.ceph:role: osd.3 2024-03-25T18:12:44.943 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:12:44.943 DEBUG:teuthology.orchestra.run.smithi132:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:12:45.019 INFO:teuthology.orchestra.run.smithi132.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:45.019 INFO:teuthology.orchestra.run.smithi132.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:45.019 INFO:teuthology.orchestra.run.smithi132.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:45.020 INFO:teuthology.orchestra.run.smithi132.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:45.020 INFO:teuthology.orchestra.run.smithi132.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:45.020 INFO:teuthology.orchestra.run.smithi132.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:45.020 INFO:teuthology.orchestra.run.smithi132.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:45.544 INFO:teuthology.orchestra.run.smithi132.stdout:Discarding blocks...Done. 2024-03-25T18:12:45.546 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi132.front.sepia.ceph.com -o noatime 2024-03-25T18:12:45.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:12:45.690 DEBUG:teuthology.orchestra.run.smithi132:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:12:45.726 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:45.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:45.826+0000 7f0543c21640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:12:45.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:45.826+0000 7f0543c21640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:12:45.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:45.826+0000 7f0543c21640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:12:45.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:45.826+0000 7f0543c21640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:12:45.827 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:45.826+0000 7f0543c21640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:12:45.828 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:45.827+0000 7f0543c21640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:12:48.674 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:12:48.704 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:48.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:48.794+0000 7fd81aa0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:12:48.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:48.794+0000 7fd81aa0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:12:48.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:48.794+0000 7fd81aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:12:48.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:48.794+0000 7fd81aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:12:48.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:48.794+0000 7fd81aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:12:48.796 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:48.795+0000 7fd81aa0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:12:51.636 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:12:51.666 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:51.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:51.756+0000 7f2009e18640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:12:51.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:51.756+0000 7f2009e18640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:12:51.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:51.756+0000 7f2009e18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:12:51.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:51.756+0000 7f2009e18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:12:51.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:51.756+0000 7f2009e18640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:12:51.757 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:51.757+0000 7f2009e18640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:12:54.596 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:12:54.628 DEBUG:teuthology.orchestra.run.smithi132:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:12:54.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:54.718+0000 7f0ccd6b1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:12:54.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:54.718+0000 7f0ccd6b1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:12:54.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:54.718+0000 7f0ccd6b1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:12:54.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:54.718+0000 7f0ccd6b1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:12:54.719 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:54.718+0000 7f0ccd6b1640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:12:54.720 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:12:54.719+0000 7f0ccd6b1640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:12:57.566 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:12:57.597 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:57.630 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:57.630 INFO:tasks.ceph:role: osd.4 2024-03-25T18:12:57.630 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:12:57.630 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:12:57.708 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:57.708 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:57.708 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:57.708 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:57.708 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:57.709 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:57.709 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:57.709 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:57.709 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:57.709 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:58.209 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-25T18:12:58.211 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-25T18:12:58.211 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:58.353 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:12:58.390 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:58.460 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:58.460 INFO:tasks.ceph:role: osd.5 2024-03-25T18:12:58.460 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:12:58.460 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:12:58.536 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:58.536 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:58.536 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:58.537 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:58.990 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-25T18:12:58.992 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-25T18:12:58.993 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:59.093 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:12:59.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:59.198 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:12:59.198 INFO:tasks.ceph:role: osd.6 2024-03-25T18:12:59.198 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:12:59.198 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:12:59.275 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:12:59.780 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-25T18:12:59.783 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-25T18:12:59.783 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:59.919 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:12:59.953 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:00.023 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:00.023 INFO:tasks.ceph:role: osd.7 2024-03-25T18:13:00.023 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:13:00.023 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:00.099 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:00.099 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:00.100 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:00.598 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-25T18:13:00.600 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-25T18:13:00.600 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:00.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:00.761 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:00.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:00.868+0000 7f7f73a12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:13:00.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:00.869+0000 7f7f73a12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:13:00.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:00.869+0000 7f7f73a12640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:00.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:00.869+0000 7f7f73a12640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:00.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:00.869+0000 7f7f73a12640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:00.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:00.869+0000 7f7f73a12640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:13:03.711 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:03.742 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:03.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:03.833+0000 7f5c1ca1c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:13:03.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:03.833+0000 7f5c1ca1c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:13:03.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:03.833+0000 7f5c1ca1c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:03.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:03.833+0000 7f5c1ca1c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:03.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:03.833+0000 7f5c1ca1c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:03.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:03.834+0000 7f5c1ca1c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:13:06.675 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:06.706 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:06.797+0000 7f8e6120d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:13:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:06.798+0000 7f8e6120d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:13:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:06.798+0000 7f8e6120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:06.798+0000 7f8e6120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:06.798+0000 7f8e6120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:06.798+0000 7f8e6120d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:13:09.639 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:09.670 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:09.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:09.760+0000 7f347da0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:13:09.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:09.760+0000 7f347da0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:13:09.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:09.760+0000 7f347da0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:09.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:09.760+0000 7f347da0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:09.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:09.760+0000 7f347da0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:09.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:09.761+0000 7f347da0d640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:13:12.602 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:12.632 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:13:12.632 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:12.633 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:13:12.664 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:12.664 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:13:12.735 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:12.735 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:13:12.805 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:12.805 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:13:12.874 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:12.874 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:13:12.943 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:12.943 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:13:12.975 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:12.976 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:13:13.045 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:13.045 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:13:13.114 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:13.115 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:13:13.183 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:13.183 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:13:13.251 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:13.252 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:13:13.268 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:13.268 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:13:13.307 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:13:13.307 DEBUG:teuthology.orchestra.run.smithi132:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:13:13.310 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:13:13.350 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.y] 2024-03-25T18:13:13.350 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCavgFm/7H/FRAAtLyiNRSWh0AWQrvYDIKfzw== 2024-03-25T18:13:13.350 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.0] 2024-03-25T18:13:13.350 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCdvgFmvk1VMRAApdJD3yEdONUqaE+x9sodPw== 2024-03-25T18:13:13.350 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.1] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCgvgFmuRVqLxAAfbiIXLYlwnkSzmaDEL1Peg== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.2] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCjvgFmAqEiLRAAR5Z2tNZTnlNNeP9hiUOPBQ== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.3] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCmvgFmnqzkKhAA+zDZBswjAHMS5BbQtFvCOg== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[mgr.x] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCavgFmwmn1GRAADe/PwJVn6eQ/qR3QvmeGHA== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.4] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCsvgFmOF3TMxAA/7g5v9mi4jFxQ9YjuwhwvA== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.5] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCvvgFmdHa7MRAAzZp9oD3QF941c7JHl5FiCw== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.6] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCyvgFmHuaYLxAAygYK9Ras/tTR71yuzfoIAg== 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout:[osd.7] 2024-03-25T18:13:13.351 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQC1vgFmQ01dLRAAfh7JWURl8TZihJNFwQcH5g== 2024-03-25T18:13:13.352 INFO:teuthology.orchestra.run.smithi132.stdout:[client.0] 2024-03-25T18:13:13.352 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCavgFmqaKjHBAATBEdt87xFsMm3f+ULy8BYQ== 2024-03-25T18:13:13.352 INFO:teuthology.orchestra.run.smithi132.stdout:[client.1] 2024-03-25T18:13:13.352 INFO:teuthology.orchestra.run.smithi132.stdout: key = AQCavgFmvm0HIBAAwRqxpNU09oVDLxiDoBLMzg== 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.y] 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCavgFm/7H/FRAAtLyiNRSWh0AWQrvYDIKfzw== 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCdvgFmvk1VMRAApdJD3yEdONUqaE+x9sodPw== 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCgvgFmuRVqLxAAfbiIXLYlwnkSzmaDEL1Peg== 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2024-03-25T18:13:13.373 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCjvgFmAqEiLRAAR5Z2tNZTnlNNeP9hiUOPBQ== 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.3] 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCmvgFmnqzkKhAA+zDZBswjAHMS5BbQtFvCOg== 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCavgFmwmn1GRAADe/PwJVn6eQ/qR3QvmeGHA== 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.4] 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCsvgFmOF3TMxAA/7g5v9mi4jFxQ9YjuwhwvA== 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.5] 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCvvgFmdHa7MRAAzZp9oD3QF941c7JHl5FiCw== 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.6] 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCyvgFmHuaYLxAAygYK9Ras/tTR71yuzfoIAg== 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.7] 2024-03-25T18:13:13.374 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQC1vgFmQ01dLRAAfh7JWURl8TZihJNFwQcH5g== 2024-03-25T18:13:13.375 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-25T18:13:13.375 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCavgFmqaKjHBAATBEdt87xFsMm3f+ULy8BYQ== 2024-03-25T18:13:13.375 INFO:teuthology.orchestra.run.smithi161.stdout:[client.1] 2024-03-25T18:13:13.375 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQCavgFmvm0HIBAAwRqxpNU09oVDLxiDoBLMzg== 2024-03-25T18:13:13.376 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:13.395 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:13.468 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.470 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.522 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.576 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.630 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.632 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.684 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:13.686 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:13.737 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.739 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.791 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.844 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.847 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.898 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.900 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:13.951 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:13.953 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:14.008 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:14.010 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:14.064 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:13:14.065 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:13:14.091 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:14.187 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:13:14.214 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:13:14.281 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:14.378 DEBUG:teuthology.orchestra.run.smithi132:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:13:14.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:13:14.432 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:14.529 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:13:14.556 DEBUG:teuthology.orchestra.run.smithi132:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:14.558 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:14.611 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:13:14.611 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:14.611 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:13:14.611 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:13:14.613 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:13:14.613 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:14.614 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:13:14.614 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:13:14.616 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:13:14.616 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:14.616 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:13:14.616 DEBUG:teuthology.orchestra.run.smithi161:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:13:14.654 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:13:14.654 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:13:14.654 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:13:14.654 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:13:14.656 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:13:14.657 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:13:14.657 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:13:14.659 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:13:14.659 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:13:14.659 DEBUG:teuthology.orchestra.run.smithi132:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:13:14.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.759+0000 7f6f86fcf640 10 monclient: get_monmap_and_config 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.759+0000 7f6f86fcf640 10 monclient: build_initial_monmap 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.759+0000 7f6f86fcf640 10 monclient: monmap: 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 0 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:13:14.760428+0000 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:13:14.760428+0000 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:14.760 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 1 Processor -- start 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 1 -- start start 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient: init 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f6f800b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:14.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f6f80142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800a9d60 con 0x7f6f80142a20 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8013ff10 con 0x7f6f800a96b0 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800aa1f0 con 0x7f6f800b4700 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f86fcf640 10 monclient(hunting): authenticate will time out at 7235.023542s 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f80142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f8013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:14.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.760+0000 7f6f8556c640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f8013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:14.961 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.961+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f8013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:14.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.961+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f80142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:14.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:14.961+0000 7f6f8556c640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f8013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:15.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:15.362+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f8013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:15.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:15.362+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f80142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:15.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:15.362+0000 7f6f8556c640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f8013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:16.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:16.163+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f80142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:16.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:16.163+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f8013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:16.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:16.163+0000 7f6f8556c640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f8013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): tick 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f80142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f8013f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f8013f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:17.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f6f80142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f6f800b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800b4ae0 con 0x7f6f80142a20 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800a9d60 con 0x7f6f800a96b0 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8013ff10 con 0x7f6f800b4700 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7e7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:17.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.760+0000 7f6f84d6b640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:17.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.961+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:17.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.962+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:17.962 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:17.962+0000 7f6f84d6b640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:18.362 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:18.362+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:18.363+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:18.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:18.363+0000 7f6f84d6b640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:19.163 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:19.163+0000 7f6f7ffff640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:19.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:19.164+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:19.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:19.164+0000 7f6f84d6b640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): tick 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:20.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f6f80142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f6f800b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800aa1f0 con 0x7f6f80142a20 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800b4ae0 con 0x7f6f800a96b0 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800a9d60 con 0x7f6f800b4700 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7e7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:20.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.762+0000 7f6f7ffff640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:20.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.963+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:20.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.963+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:20.963 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:20.963+0000 7f6f7ffff640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:21.363 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:21.364+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:21.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:21.364+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:21.364 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:21.364+0000 7f6f7ffff640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:21.622 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:00:06.863 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:21.928 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:00:07.169 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:21.962 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:00:07.226 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:21.962 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:00:07.227 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:22.164 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:22.165+0000 7f6f84d6b640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6789/0 2024-03-25T18:13:22.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:22.165+0000 7f6f8556c640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.132:6790/0 2024-03-25T18:13:22.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:22.165+0000 7f6f7ffff640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.161:6789/0 2024-03-25T18:13:22.282 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:00:07.546 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:22.283 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:00:07.547 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): tick 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 -- >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f64008e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:23.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f6f80142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:23.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f6f800b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:23.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:23.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8013ff10 con 0x7f6f80142a20 2024-03-25T18:13:23.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800aa1f0 con 0x7f6f800a96b0 2024-03-25T18:13:23.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800b4ae0 con 0x7f6f800b4700 2024-03-25T18:13:23.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.761+0000 7f6f7e7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:23.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.850+0000 7f6f8556c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 0x7f6f640056d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:51822/0 (socket says 172.21.15.132:51822) 2024-03-25T18:13:23.851 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:23.850+0000 7f6f8556c640 1 -- 172.21.15.132:0/1451887525 learned_addr learned my addr 172.21.15.132:0/1451887525 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:26.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): tick 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.132:6789/0 conn(0x7f6f80142a20 legacy=0x7f6f6400cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f6f64008ef0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f800a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f6f800b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800a9d60 con 0x7f6f64008ef0 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f64009330 con 0x7f6f800a96b0 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6400cf90 con 0x7f6f800b4700 2024-03-25T18:13:26.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:26.761+0000 7f6f7e7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:29.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): tick 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.132:6790/0 conn(0x7f6f800b4700 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.161:6789/0 conn(0x7f6f800a96b0 legacy=0x7f6f640056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.132:6789/0 conn(0x7f6f64008ef0 legacy=0x7f6f6400d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f6f6400df60 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f6f64010ac0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f6f64008ef0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f6400d1c0 con 0x7f6f64008ef0 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f640066e0 con 0x7f6f64010ac0 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f64007880 con 0x7f6f6400df60 2024-03-25T18:13:29.763 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:29.761+0000 7f6f7e7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:32.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.222+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382473977 0 0) 0x7f6f64007880 con 0x7f6f6400df60 2024-03-25T18:13:32.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.222+0000 7f6f7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.222+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.222+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:32.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.222+0000 7f6f7f7fe640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:13:32.223 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.222+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f70003520 con 0x7f6f6400df60 2024-03-25T18:13:32.249 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.249+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091332144 0 0) 0x7f6f640066e0 con 0x7f6f64010ac0 2024-03-25T18:13:32.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.249+0000 7f6f7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.249+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.249+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:32.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.249+0000 7f6f7f7fe640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:13:32.250 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.249+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f64007880 con 0x7f6f64010ac0 2024-03-25T18:13:32.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.263+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767771926 0 0) 0x7f6f6400d1c0 con 0x7f6f64008ef0 2024-03-25T18:13:32.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.263+0000 7f6f7f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.263+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.263+0000 7f6f7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:32.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.263+0000 7f6f7f7fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:13:32.264 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.263+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f640066e0 con 0x7f6f64008ef0 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2903389584 0 0) 0x7f6f70003520 con 0x7f6f6400df60 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.161:6789/0 conn(0x7f6f64010ac0 legacy=0x7f6f6400c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.132:6789/0 conn(0x7f6f64008ef0 legacy=0x7f6f6400d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f64008a70 con 0x7f6f6400df60 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:32.283 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6f74003f60 con 0x7f6f6400df60 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f86fcf640 5 monclient: authenticate success, global_id 4103 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f86fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f7f7fe640 10 monclient: dump: 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:32.284 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:32.285 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:32.285 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:32.285 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:32.285 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.285 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.282+0000 7f6f86fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f740040c0 con 0x7f6f6400df60 2024-03-25T18:13:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f7f7fe640 1 -- 172.21.15.132:0/1451887525 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6f74004510 con 0x7f6f6400df60 2024-03-25T18:13:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:32.321 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f7f7fe640 10 monclient: dump: 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f86fcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:32.322 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.320+0000 7f6f86fcf640 4 set_mon_vals no callback set 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 10 monclient: shutdown 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1451887525 >> v1:172.21.15.132:6790/0 conn(0x7f6f6400df60 legacy=0x7f6f640103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1451887525 shutdown_connections 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1451887525 >> 172.21.15.132:0/1451887525 conn(0x7f6f800668e0 msgr2=0x7f6f80066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1451887525 shutdown_connections 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.321+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1451887525 wait complete. 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.322+0000 7f6f86fcf640 10 monclient: build_initial_monmap 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.322+0000 7f6f86fcf640 10 monclient: monmap: 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 0 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 0.000000 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:created 0.000000 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:32.324 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 1 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 librados: starting msgr at 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 librados: starting objecter 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 Processor -- start 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 -- start start 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 librados: setting wanted keys 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 librados: calling monclient init 2024-03-25T18:13:32.325 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient: init 2024-03-25T18:13:32.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:32.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:32.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f6f64008ef0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:32.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f6f6400df60 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:32.326 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f6f64010ac0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8013a490 con 0x7f6f6400df60 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800b1180 con 0x7f6f64008ef0 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f800b16a0 con 0x7f6f64010ac0 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.323+0000 7f6f86fcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.324+0000 7f6f86fcf640 10 monclient(hunting): authenticate will time out at 7252.586717s 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.324+0000 7f6f7ffff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f6f6400df60 0x7f6f80139670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:60998/0 (socket says 172.21.15.132:60998) 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.324+0000 7f6f7ffff640 1 -- 172.21.15.132:0/1732838520 learned_addr learned my addr 172.21.15.132:0/1732838520 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.324+0000 7f6f7d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f6400df60 2024-03-25T18:13:32.327 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.324+0000 7f6f7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:32.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.328+0000 7f6f7d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f64010ac0 2024-03-25T18:13:32.329 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.328+0000 7f6f7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:32.331 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.331+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876606369 0 0) 0x7f6f800b16a0 con 0x7f6f64010ac0 2024-03-25T18:13:32.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.331+0000 7f6f7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.331+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.331+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:32.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.331+0000 7f6f7d7fa640 10 monclient(hunting): my global_id is 4109 2024-03-25T18:13:32.332 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.331+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f6400d1c0 con 0x7f6f64010ac0 2024-03-25T18:13:32.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.340+0000 7f6f7d7fa640 10 client.?.objecter ms_handle_connect 0x7f6f64008ef0 2024-03-25T18:13:32.341 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.340+0000 7f6f7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:32.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.347+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1779027782 0 0) 0x7f6f6400d1c0 con 0x7f6f64010ac0 2024-03-25T18:13:32.348 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.347+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f800b16a0 con 0x7f6f64010ac0 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.348+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6f78004560 con 0x7f6f64010ac0 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.348+0000 7f6f7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.348+0000 7f6f7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.348+0000 7f6f7d7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:32.349 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.350 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.348+0000 7f6f7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:32.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.384+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468529379 0 0) 0x7f6f8013a490 con 0x7f6f6400df60 2024-03-25T18:13:32.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.384+0000 7f6f7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.384+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.384+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:32.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.384+0000 7f6f7d7fa640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:13:32.385 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.384+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f6400d1c0 con 0x7f6f6400df60 2024-03-25T18:13:32.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.385+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669070164 0 0) 0x7f6f800b1180 con 0x7f6f64008ef0 2024-03-25T18:13:32.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.385+0000 7f6f7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.385+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.385+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:32.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.385+0000 7f6f7d7fa640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:13:32.386 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.385+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f8013a490 con 0x7f6f64008ef0 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.409+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2267670488 0 0) 0x7f6f8013a490 con 0x7f6f64008ef0 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.409+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f800b1180 con 0x7f6f64008ef0 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.409+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6f74004410 con 0x7f6f64008ef0 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.409+0000 7f6f7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.409+0000 7f6f7d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.409+0000 7f6f7d7fa640 10 monclient(hunting): dump: 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:32.410 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:32.411 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3690418849 0 0) 0x7f6f800b16a0 con 0x7f6f64010ac0 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.161:6789/0 conn(0x7f6f64008ef0 legacy=0x7f6f801d7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6789/0 conn(0x7f6f6400df60 legacy=0x7f6f80139670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 1 monclient: found mon.c 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8013ac00 con 0x7f6f64010ac0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 5 monclient: authenticate success, global_id 4109 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 10 monclient: _renew_subs 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8013ae50 con 0x7f6f64010ac0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 10 monclient: _renew_subs 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f801d8260 con 0x7f6f64010ac0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.411+0000 7f6f86fcf640 1 librados: init done 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.412+0000 7f6f86fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.412+0000 7f6f86fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.412+0000 7f6f86fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.412+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f80120d40 con 0x7f6f64010ac0 2024-03-25T18:13:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f8556c640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64010ac0 legacy=0x7f6f80139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:13:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f8556c640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64010ac0 legacy=0x7f6f80139d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:13:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f8556c640 1 --1- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64010ac0 0x7f6f80139d80 :-1 s=OPENED pgs=16 cs=1 l=1).handle_message read tag failed 2024-03-25T18:13:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f8556c640 1 --1- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64010ac0 0x7f6f80139d80 :-1 s=OPENED pgs=16 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:13:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.132:6790/0 2024-03-25T18:13:32.421 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64010ac0 legacy=0x7f6f80139d80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient(hunting): picked mon.b con 0x7f6f64008ef0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient(hunting): picked mon.c con 0x7f6f64008600 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient(hunting): picked mon.a con 0x7f6f6400df60 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6f800b16a0 con 0x7f6f6400df60 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6f8013a490 con 0x7f6f64008ef0 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6f64005d20 con 0x7f6f64008600 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.420+0000 7f6f7d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:32.422 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.421+0000 7f6f7d7fa640 10 client.4109.objecter ms_handle_connect 0x7f6f64008ef0 2024-03-25T18:13:32.423 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.421+0000 7f6f7d7fa640 10 client.4109.objecter resend_mon_ops 2024-03-25T18:13:32.439 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.439+0000 7f6f7d7fa640 10 client.4109.objecter ms_handle_connect 0x7f6f64008600 2024-03-25T18:13:32.440 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.439+0000 7f6f7d7fa640 10 client.4109.objecter resend_mon_ops 2024-03-25T18:13:32.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.470+0000 7f6f7d7fa640 10 client.4109.objecter ms_handle_connect 0x7f6f6400df60 2024-03-25T18:13:32.471 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.470+0000 7f6f7d7fa640 10 client.4109.objecter resend_mon_ops 2024-03-25T18:13:32.494 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.494+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106034238 0 0) 0x7f6f64005d20 con 0x7f6f64008600 2024-03-25T18:13:32.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.494+0000 7f6f7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.494+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.494+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:13:32.495 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.494+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f6f640063e0 con 0x7f6f64008600 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2406562905 0 0) 0x7f6f640063e0 con 0x7f6f64008600 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.161:6789/0 conn(0x7f6f64008ef0 legacy=0x7f6f64002b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6789/0 conn(0x7f6f6400df60 legacy=0x7f6f64005610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 monclient: found mon.c 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f6f640060d0 con 0x7f6f64008600 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:32.515 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f64005d20 con 0x7f6f64008600 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6f6c003f30 con 0x7f6f64008600 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.514+0000 7f6f7d7fa640 10 monclient: dump: 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:32.516 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:32.517 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:32.517 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f6c004090 con 0x7f6f64008600 2024-03-25T18:13:32.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:32.737 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7ffff640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64008600 legacy=0x7f6f640095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7ffff640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64008600 legacy=0x7f6f640095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7ffff640 1 --1- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64008600 0x7f6f640095b0 :-1 s=OPENED pgs=19 cs=1 l=1).handle_message read tag failed 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7ffff640 1 --1- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64008600 0x7f6f640095b0 :-1 s=OPENED pgs=19 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6f6c004690 con 0x7f6f64008600 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7d7fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.132:6790/0 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7d7fa640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64008600 legacy=0x7f6f640095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:13:32.738 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.736+0000 7f6f7effd640 4 set_mon_vals no callback set 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 10 monclient(hunting): picked mon.c con 0x7f6f64010ac0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 10 monclient(hunting): picked mon.a con 0x7f6f6400df60 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 10 monclient(hunting): picked mon.b con 0x7f6f64008ef0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6f640063e0 con 0x7f6f6400df60 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6f64019d20 con 0x7f6f64008ef0 2024-03-25T18:13:32.739 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f6f6401aec0 con 0x7f6f64010ac0 2024-03-25T18:13:32.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.737+0000 7f6f7d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:32.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.738+0000 7f6f7d7fa640 10 client.4109.objecter ms_handle_connect 0x7f6f6400df60 2024-03-25T18:13:32.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.738+0000 7f6f7d7fa640 10 client.4109.objecter resend_mon_ops 2024-03-25T18:13:32.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.738+0000 7f6f7d7fa640 10 client.4109.objecter ms_handle_connect 0x7f6f64008ef0 2024-03-25T18:13:32.740 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.738+0000 7f6f7d7fa640 10 client.4109.objecter resend_mon_ops 2024-03-25T18:13:32.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.747+0000 7f6f7d7fa640 10 client.4109.objecter ms_handle_connect 0x7f6f64010ac0 2024-03-25T18:13:32.748 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.747+0000 7f6f7d7fa640 10 client.4109.objecter resend_mon_ops 2024-03-25T18:13:32.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.788+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3466736177 0 0) 0x7f6f6401aec0 con 0x7f6f64010ac0 2024-03-25T18:13:32.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.788+0000 7f6f7d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:32.788 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.788+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:32.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.788+0000 7f6f7d7fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:13:32.789 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.788+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f6f640095b0 con 0x7f6f64010ac0 2024-03-25T18:13:32.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2847953728 0 0) 0x7f6f640095b0 con 0x7f6f64010ac0 2024-03-25T18:13:32.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.161:6789/0 conn(0x7f6f64008ef0 legacy=0x7f6f64002b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.793 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6789/0 conn(0x7f6f6400df60 legacy=0x7f6f64005610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 monclient: found mon.c 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f6f6401c060 con 0x7f6f64010ac0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f6401aec0 con 0x7f6f64010ac0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6f74004860 con 0x7f6f64010ac0 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:32.794 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.792+0000 7f6f7d7fa640 10 monclient: dump: 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:32.795 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:32.896 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:32.895+0000 7f2a02236040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:13:32.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f740049c0 con 0x7f6f64010ac0 2024-03-25T18:13:32.901 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6f74004ec0 con 0x7f6f64010ac0 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 6 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1137662361 0 0) 0x7f6f74006340 con 0x7f6f64010ac0 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 10 client.4109.objecter ms_dispatch 0x7f6f80000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 3 client.4109.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 3 client.4109.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7e7fc640 4 set_mon_vals no callback set 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6f740066e0 con 0x7f6f64010ac0 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:32.902 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.901+0000 7f6f7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:32.950 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:32.950+0000 7f2a02236040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:13:32.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.969+0000 7f6f86fcf640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:13:32.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.969+0000 7f6f86fcf640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:13:32.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.969+0000 7f6f86fcf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:32.970 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:32.969+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6f800b0fe0 con 0x7f6f64010ac0 2024-03-25T18:13:33.029 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.028+0000 7fd45a62d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:13:33.105 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.104+0000 7fd45a62d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:13:33.165 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f7d7fa640 1 -- 172.21.15.132:0/1732838520 <== mon.2 v1:172.21.15.132:6790/0 8 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f6f800b0fe0 con 0x7f6f64010ac0 2024-03-25T18:13:33.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:13:33.166 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f7d7fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:13:33.166 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f86fcf640 10 librados: watch_flush enter 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f86fcf640 10 librados: watch_flush exit 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f86fcf640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.165+0000 7f6f86fcf640 10 client.4109.objecter successfully canceled tick 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 10 monclient: shutdown 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 >> v1:172.21.15.132:6790/0 conn(0x7f6f64010ac0 legacy=0x7f6f64017280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.167 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 shutdown_connections 2024-03-25T18:13:33.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 >> 172.21.15.132:0/1732838520 conn(0x7f6f800668e0 msgr2=0x7f6f800b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:33.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 shutdown_connections 2024-03-25T18:13:33.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 1 -- 172.21.15.132:0/1732838520 wait complete. 2024-03-25T18:13:33.168 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:33.166+0000 7f6f86fcf640 1 librados: shutdown 2024-03-25T18:13:33.177 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:13:33.177 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:13:33.177 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:33.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:13:33.205 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:33.205 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:13:33.275 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:33.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:13:33.342 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2024-03-25T18:13:33.342 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:13:33.409 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:33.409 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:13:33.436 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:33.437 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:13:33.504 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:33.504 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:13:33.516 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.516+0000 7f2a02236040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:13:33.543 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-25T18:13:33.543 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:13:33.574 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.573+0000 7f2a02236040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:13:33.600 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 9a272c8b-fb39-4075-b438-a2a51b258a70 0 2024-03-25T18:13:33.621 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.620+0000 7f2a02236040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:13:33.678 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.677+0000 7f2a02236040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:13:33.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.720+0000 7fa353167640 10 monclient: get_monmap_and_config 2024-03-25T18:13:33.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.720+0000 7fa353167640 10 monclient: build_initial_monmap 2024-03-25T18:13:33.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.720+0000 7fa353167640 10 monclient: monmap: 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:33.720881+0000 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:33.720881+0000 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:33.721 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.720+0000 7fa353167640 1 Processor -- start 2024-03-25T18:13:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.720+0000 7fa353167640 1 -- start start 2024-03-25T18:13:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.720+0000 7fa353167640 10 monclient: init 2024-03-25T18:13:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient(hunting): picked mon.noname-a con 0x7fa34c0b4700 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient(hunting): picked mon.noname-c con 0x7fa34c142a20 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient(hunting): picked mon.noname-b con 0x7fa34c0a96b0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0a9d60 con 0x7fa34c0b4700 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c13ff10 con 0x7fa34c142a20 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c0aa1f0 con 0x7fa34c0a96b0 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.721+0000 7fa353167640 10 monclient(hunting): authenticate will time out at 7257.051259s 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.722+0000 7fa350f03640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa34c0b4700 0x7fa34c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52610/0 (socket says 172.21.15.161:52610) 2024-03-25T18:13:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.722+0000 7fa350f03640 1 -- 172.21.15.161:0/1180699063 learned_addr learned my addr 172.21.15.161:0/1180699063 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:33.723 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.722+0000 7f2a02236040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:13:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927207279 0 0) 0x7fa34c0aa1f0 con 0x7fa34c0a96b0 2024-03-25T18:13:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:13:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c003520 con 0x7fa34c0a96b0 2024-03-25T18:13:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343372436 0 0) 0x7fa34c13ff10 con 0x7fa34c142a20 2024-03-25T18:13:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.724+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34c0aa1f0 con 0x7fa34c142a20 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.725+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271022308 0 0) 0x7fa34c0a9d60 con 0x7fa34c0b4700 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.725+0000 7fa34b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.725+0000 7fa34b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.725+0000 7fa34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.725+0000 7fa34b7fe640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:13:33.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.725+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34c13ff10 con 0x7fa34c0b4700 2024-03-25T18:13:33.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700170501 0 0) 0x7fa33c003520 con 0x7fa34c0a96b0 2024-03-25T18:13:33.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 >> v1:172.21.15.161:6789/0 conn(0x7fa34c142a20 legacy=0x7fa34c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 >> v1:172.21.15.132:6789/0 conn(0x7fa34c0b4700 legacy=0x7fa34c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:33.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c0b4ae0 con 0x7fa34c0a96b0 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa344001a10 con 0x7fa34c0a96b0 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa34b7fe640 10 monclient: dump: 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.729 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa353167640 5 monclient: authenticate success, global_id 4118 2024-03-25T18:13:33.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.727+0000 7fa353167640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa344001b70 con 0x7fa34c0a96b0 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa34b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa34b7fe640 1 -- 172.21.15.161:0/1180699063 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa344001f80 con 0x7fa34c0a96b0 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 4 set_mon_vals no callback set 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa34b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa34b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:33.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa34b7fe640 10 monclient: dump: 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 10 monclient: shutdown 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 1 -- 172.21.15.161:0/1180699063 >> v1:172.21.15.132:6790/0 conn(0x7fa34c0a96b0 legacy=0x7fa34c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 1 -- 172.21.15.161:0/1180699063 shutdown_connections 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 1 -- 172.21.15.161:0/1180699063 >> 172.21.15.161:0/1180699063 conn(0x7fa34c0668e0 msgr2=0x7fa34c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 1 -- 172.21.15.161:0/1180699063 shutdown_connections 2024-03-25T18:13:33.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.730+0000 7fa353167640 1 -- 172.21.15.161:0/1180699063 wait complete. 2024-03-25T18:13:33.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 10 monclient: build_initial_monmap 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 10 monclient: monmap: 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 1 librados: starting msgr at 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 1 librados: starting objecter 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 1 Processor -- start 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 1 -- start start 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 1 librados: setting wanted keys 2024-03-25T18:13:33.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 1 librados: calling monclient init 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.731+0000 7fa353167640 10 monclient: init 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient(hunting): picked mon.noname-a con 0x7fa34c0a96b0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient(hunting): picked mon.noname-c con 0x7fa34c0b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient(hunting): picked mon.noname-b con 0x7fa34c142a20 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c13b330 con 0x7fa34c0a96b0 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c130b80 con 0x7fa34c142a20 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34c13a240 con 0x7fa34c0b4700 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.732+0000 7fa353167640 10 monclient(hunting): authenticate will time out at 7257.062350s 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.733+0000 7fa351704640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa34c142a20 0x7fa34c130470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51368/0 (socket says 172.21.15.161:51368) 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.733+0000 7fa351704640 1 -- 172.21.15.161:0/2486535918 learned_addr learned my addr 172.21.15.161:0/2486535918 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa3497fa640 10 client.?.objecter ms_handle_connect 0x7fa34c142a20 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa3497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:33.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa3497fa640 10 client.?.objecter ms_handle_connect 0x7fa34c0a96b0 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa3497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa350f03640 1 -- 172.21.15.161:0/2486535918 >> v1:172.21.15.132:6789/0 conn(0x7fa34c0a96b0 legacy=0x7fa34c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa350f03640 1 -- 172.21.15.161:0/2486535918 >> v1:172.21.15.132:6789/0 conn(0x7fa34c0a96b0 legacy=0x7fa34c13ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa350f03640 1 --1- 172.21.15.161:0/2486535918 >> v1:172.21.15.132:6789/0 conn(0x7fa34c0a96b0 0x7fa34c13ac20 :-1 s=OPENED pgs=21 cs=1 l=1).handle_message read tag failed 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa350f03640 1 --1- 172.21.15.161:0/2486535918 >> v1:172.21.15.132:6789/0 conn(0x7fa34c0a96b0 0x7fa34c13ac20 :-1 s=OPENED pgs=21 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.734+0000 7fa3497fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.132:6789/0 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.735+0000 7fa3497fa640 10 client.?.objecter ms_handle_connect 0x7fa34c0b4700 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.735+0000 7fa3497fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:33.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.736+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713183923 0 0) 0x7fa34c130b80 con 0x7fa34c142a20 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.736+0000 7fa3497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.736+0000 7fa3497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.736+0000 7fa3497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.736+0000 7fa3497fa640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.736+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa330004eb0 con 0x7fa34c142a20 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.737+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1674850320 0 0) 0x7fa34c13a240 con 0x7fa34c0b4700 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.737+0000 7fa3497fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.737+0000 7fa3497fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.737+0000 7fa3497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.737+0000 7fa3497fa640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:13:33.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.737+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa34c130b80 con 0x7fa34c0b4700 2024-03-25T18:13:33.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 253792197 0 0) 0x7fa330004eb0 con 0x7fa34c142a20 2024-03-25T18:13:33.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa34c13a240 con 0x7fa34c142a20 2024-03-25T18:13:33.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa344001ec0 con 0x7fa34c142a20 2024-03-25T18:13:33.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 10 monclient(hunting): dump: 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.741+0000 7fa3497fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:33.743 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.741+0000 7fd45a62d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.742+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2370690709 0 0) 0x7fa34c130b80 con 0x7fa34c0b4700 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.742+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa330004eb0 con 0x7fa34c0b4700 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.742+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa3380017d0 con 0x7fa34c0b4700 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.742+0000 7fa3497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.742+0000 7fa3497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.742+0000 7fa3497fa640 10 monclient(hunting): dump: 2024-03-25T18:13:33.743 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:33.744 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3346473263 0 0) 0x7fa330004eb0 con 0x7fa34c0b4700 2024-03-25T18:13:33.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 >> v1:172.21.15.161:6789/0 conn(0x7fa34c142a20 legacy=0x7fa34c130470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 >> v1:172.21.15.132:6789/0 conn(0x7fa34c0a96b0 legacy=0x7fa34c13ac20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 1 monclient: found mon.c 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.745+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34c13a520 con 0x7fa34c0b4700 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa3497fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa3497fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa3497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 monclient: _renew_subs 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34c13a800 con 0x7fa34c0b4700 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 monclient: _renew_subs 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:33.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34c142280 con 0x7fa34c0b4700 2024-03-25T18:13:33.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 1 librados: init done 2024-03-25T18:13:33.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:33.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:33.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:33.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.746+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34c04f730 con 0x7fa34c0b4700 2024-03-25T18:13:33.799 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.798+0000 7fd45a62d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:13:33.843 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.842+0000 7f2a02236040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:13:33.845 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.844+0000 7fd45a62d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa338001d00 con 0x7fa34c0b4700 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa3380037b0 con 0x7fa34c0b4700 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 10 monclient: dump: 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:33.847 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:33.848 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:33.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa338003ce0 con 0x7fa34c0b4700 2024-03-25T18:13:33.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.846+0000 7fa34affd640 4 set_mon_vals no callback set 2024-03-25T18:13:33.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 577445391 0 0) 0x7fa338005220 con 0x7fa34c0b4700 2024-03-25T18:13:33.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 10 client.4121.objecter ms_dispatch 0x7fa34c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:13:33.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 3 client.4121.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:13:33.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 3 client.4121.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:13:33.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T18:13:33.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa338004230 con 0x7fa34c0b4700 2024-03-25T18:13:33.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:33.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.847+0000 7fa3497fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:33.898 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.897+0000 7fd45a62d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:13:33.915 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.914+0000 7f2a02236040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:13:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.920+0000 7fa353167640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9a272c8b-fb39-4075-b438-a2a51b258a70", "id": 0}] 2024-03-25T18:13:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.920+0000 7fa353167640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9a272c8b-fb39-4075-b438-a2a51b258a70", "id": 0}] 2024-03-25T18:13:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.920+0000 7fa353167640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:33.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:33.920+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd new", "uuid": "9a272c8b-fb39-4075-b438-a2a51b258a70", "id": 0} v 0) v1 -- 0x7fa34c1387b0 con 0x7fa34c0b4700 2024-03-25T18:13:33.941 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:33.941+0000 7fd45a62d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:13:33.961 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:33.961+0000 7f2a02236040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:13:34.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.007+0000 7fa3497fa640 1 -- 172.21.15.161:0/2486535918 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9a272c8b-fb39-4075-b438-a2a51b258a70", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2172063485 0 732707515) 0x7fa338004500 con 0x7fa34c0b4700 2024-03-25T18:13:34.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.007+0000 7fa3497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9a272c8b-fb39-4075-b438-a2a51b258a70", "id": 0}] 2024-03-25T18:13:34.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.007+0000 7fa3497fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:34.008 INFO:teuthology.orchestra.run.smithi161.stdout:0 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 10 librados: watch_flush enter 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 10 librados: watch_flush exit 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 10 client.4121.objecter successfully canceled tick 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 10 monclient: shutdown 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 >> v1:172.21.15.132:6790/0 conn(0x7fa34c0b4700 legacy=0x7fa34c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 shutdown_connections 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 >> 172.21.15.161:0/2486535918 conn(0x7fa34c0668e0 msgr2=0x7fa34c0b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 shutdown_connections 2024-03-25T18:13:34.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 1 -- 172.21.15.161:0/2486535918 wait complete. 2024-03-25T18:13:34.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.008+0000 7fa353167640 1 librados: shutdown 2024-03-25T18:13:34.016 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.016+0000 7f2a02236040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:13:34.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new eff0a134-4085-4f7d-840d-fa3e3b399d35 1 2024-03-25T18:13:34.067 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.067+0000 7f2a02236040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:13:34.083 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.082+0000 7fd45a62d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:13:34.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.119+0000 7f6b8f504640 10 monclient: get_monmap_and_config 2024-03-25T18:13:34.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.119+0000 7f6b8f504640 10 monclient: build_initial_monmap 2024-03-25T18:13:34.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.119+0000 7f6b8f504640 10 monclient: monmap: 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:34.120047+0000 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:34.120047+0000 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:34.120 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.119+0000 7f6b8f504640 1 Processor -- start 2024-03-25T18:13:34.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 1 -- start start 2024-03-25T18:13:34.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient: init 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient(hunting): picked mon.noname-a con 0x7f6b88070210 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient(hunting): picked mon.noname-c con 0x7f6b880b06d0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient(hunting): picked mon.noname-b con 0x7f6b880705f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880b0b80 con 0x7f6b88070210 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880b0430 con 0x7f6b880b06d0 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88071670 con 0x7f6b880705f0 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.120+0000 7f6b8f504640 10 monclient(hunting): authenticate will time out at 7257.450467s 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.121+0000 7f6b8d2a0640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f6b88070210 0x7f6b88140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52630/0 (socket says 172.21.15.161:52630) 2024-03-25T18:13:34.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.121+0000 7f6b8d2a0640 1 -- 172.21.15.161:0/3837254673 learned_addr learned my addr 172.21.15.161:0/3837254673 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:34.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.123+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237512709 0 0) 0x7f6b84001110 con 0x7f6b880b06d0 2024-03-25T18:13:34.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b74003520 con 0x7f6b880b06d0 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537953815 0 0) 0x7f6b880b0b80 con 0x7f6b88070210 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b84001110 con 0x7f6b88070210 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603894992 0 0) 0x7f6b88071670 con 0x7f6b880705f0 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:13:34.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.124+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b880b0b80 con 0x7f6b880705f0 2024-03-25T18:13:34.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434798408 0 0) 0x7f6b74003520 con 0x7f6b880b06d0 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 >> v1:172.21.15.132:6790/0 conn(0x7f6b880705f0 legacy=0x7f6b8814b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 >> v1:172.21.15.132:6789/0 conn(0x7f6b88070210 legacy=0x7f6b88140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 1 monclient: found mon.noname-c 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88070a60 con 0x7f6b880b06d0 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b8f504640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6b84001680 con 0x7f6b880b06d0 2024-03-25T18:13:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b8f504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b7ffff640 10 monclient: dump: 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.127+0000 7f6b8f504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:34.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b840017e0 con 0x7f6b880b06d0 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b7ffff640 1 -- 172.21.15.161:0/3837254673 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6b84001b70 con 0x7f6b880b06d0 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b7ffff640 10 monclient: dump: 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b8f504640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.129+0000 7f6b8f504640 4 set_mon_vals no callback set 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 10 monclient: shutdown 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 1 -- 172.21.15.161:0/3837254673 >> v1:172.21.15.161:6789/0 conn(0x7f6b880b06d0 legacy=0x7f6b880afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 1 -- 172.21.15.161:0/3837254673 shutdown_connections 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 1 -- 172.21.15.161:0/3837254673 >> 172.21.15.161:0/3837254673 conn(0x7f6b880b5cd0 msgr2=0x7f6b880b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 1 -- 172.21.15.161:0/3837254673 shutdown_connections 2024-03-25T18:13:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.130+0000 7f6b8f504640 1 -- 172.21.15.161:0/3837254673 wait complete. 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 10 monclient: build_initial_monmap 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 10 monclient: monmap: 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:34.134 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 1 librados: starting msgr at 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 1 librados: starting objecter 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 1 Processor -- start 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 1 -- start start 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 1 librados: setting wanted keys 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 1 librados: calling monclient init 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.131+0000 7f6b8f504640 10 monclient: init 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient(hunting): picked mon.noname-c con 0x7f6b88070210 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient(hunting): picked mon.noname-b con 0x7f6b880705f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient(hunting): picked mon.noname-a con 0x7f6b880b06d0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b881440b0 con 0x7f6b880b06d0 2024-03-25T18:13:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b881425f0 con 0x7f6b880705f0 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b881316e0 con 0x7f6b88070210 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8f504640 10 monclient(hunting): authenticate will time out at 7257.461860s 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8d2a0640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f6b88070210 0x7f6b88130fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.161:34108/0 (socket says 172.21.15.161:34108) 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.132+0000 7f6b8d2a0640 1 -- 172.21.15.161:0/2984225336 learned_addr learned my addr 172.21.15.161:0/2984225336 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.133+0000 7f6b7dffb640 10 client.?.objecter ms_handle_connect 0x7f6b88070210 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.133+0000 7f6b7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.133+0000 7f6b7dffb640 10 client.?.objecter ms_handle_connect 0x7f6b880705f0 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.133+0000 7f6b7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.133+0000 7f6b7dffb640 10 client.?.objecter ms_handle_connect 0x7f6b880b06d0 2024-03-25T18:13:34.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.133+0000 7f6b7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912151043 0 0) 0x7f6b881316e0 con 0x7f6b88070210 2024-03-25T18:13:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:13:34.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b6c004eb0 con 0x7f6b88070210 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284885756 0 0) 0x7f6b881425f0 con 0x7f6b880705f0 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b881316e0 con 0x7f6b880705f0 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2903226134 0 0) 0x7f6b881440b0 con 0x7f6b880b06d0 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:13:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.135+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b881425f0 con 0x7f6b880b06d0 2024-03-25T18:13:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3208093471 0 0) 0x7f6b881316e0 con 0x7f6b880705f0 2024-03-25T18:13:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b881440b0 con 0x7f6b880705f0 2024-03-25T18:13:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6b84000fb0 con 0x7f6b880705f0 2024-03-25T18:13:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 126643102 0 0) 0x7f6b6c004eb0 con 0x7f6b88070210 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b881316e0 con 0x7f6b88070210 2024-03-25T18:13:34.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6b780019b0 con 0x7f6b88070210 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 10 monclient(hunting): dump: 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1267851469 0 0) 0x7f6b881425f0 con 0x7f6b880b06d0 2024-03-25T18:13:34.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.140+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b6c004eb0 con 0x7f6b880b06d0 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2097315160 0 0) 0x7f6b881440b0 con 0x7f6b880705f0 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 >> v1:172.21.15.132:6790/0 conn(0x7f6b88070210 legacy=0x7f6b88130fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 >> v1:172.21.15.132:6789/0 conn(0x7f6b880b06d0 legacy=0x7f6b881d8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 1 monclient: found mon.b 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.142+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88131870 con 0x7f6b880705f0 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b7dffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:34.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b7dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 10 monclient: _renew_subs 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b881434f0 con 0x7f6b880705f0 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 10 monclient: _renew_subs 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b88143c90 con 0x7f6b880705f0 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.143+0000 7f6b8f504640 1 librados: init done 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.144+0000 7f6b8f504640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.144+0000 7f6b8f504640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.144+0000 7f6b8f504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.144+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b8804f730 con 0x7f6b880705f0 2024-03-25T18:13:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.147+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b84001370 con 0x7f6b880705f0 2024-03-25T18:13:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.147+0000 7f6b7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:34.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.147+0000 7f6b7f7fe640 4 set_mon_vals no callback set 2024-03-25T18:13:34.155 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.154+0000 7fd45a62d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:13:34.201 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.201+0000 7fd45a62d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:13:34.253 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.253+0000 7fd45a62d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:13:34.301 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.300+0000 7fd45a62d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6b84003a40 con 0x7f6b880705f0 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 10 monclient: dump: 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.334 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6b84003f30 con 0x7f6b880705f0 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3101244736 0 0) 0x7f6b84005520 con 0x7f6b880705f0 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 10 client.4126.objecter ms_dispatch 0x7f6b88000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6b84005900 con 0x7f6b880705f0 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.333+0000 7f6b7dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:34.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.432+0000 7f6b8f504640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eff0a134-4085-4f7d-840d-fa3e3b399d35", "id": 1}] 2024-03-25T18:13:34.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.432+0000 7f6b8f504640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eff0a134-4085-4f7d-840d-fa3e3b399d35", "id": 1}] 2024-03-25T18:13:34.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.432+0000 7f6b8f504640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.432+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd new", "uuid": "eff0a134-4085-4f7d-840d-fa3e3b399d35", "id": 1} v 0) v1 -- 0x7f6b880b51c0 con 0x7f6b880705f0 2024-03-25T18:13:34.463 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.463+0000 7f2a02236040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:13:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.554+0000 7f6b7dffb640 1 -- 172.21.15.161:0/2984225336 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eff0a134-4085-4f7d-840d-fa3e3b399d35", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 297291981 0 940483276) 0x7f6b84005aa0 con 0x7f6b880705f0 2024-03-25T18:13:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.554+0000 7f6b7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eff0a134-4085-4f7d-840d-fa3e3b399d35", "id": 1}] 2024-03-25T18:13:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.554+0000 7f6b7dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:34.555 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-25T18:13:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 10 librados: watch_flush enter 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 10 librados: watch_flush exit 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 10 client.4126.objecter successfully canceled tick 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 10 monclient: shutdown 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 >> v1:172.21.15.161:6789/0 conn(0x7f6b880705f0 legacy=0x7f6b881d8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 shutdown_connections 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 >> 172.21.15.161:0/2984225336 conn(0x7f6b880b5cd0 msgr2=0x7f6b880afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 shutdown_connections 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 1 -- 172.21.15.161:0/2984225336 wait complete. 2024-03-25T18:13:34.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.555+0000 7f6b8f504640 1 librados: shutdown 2024-03-25T18:13:34.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8 2 2024-03-25T18:13:34.606 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.606+0000 7f2a02236040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.666+0000 7f0363aa8640 10 monclient: get_monmap_and_config 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.666+0000 7f0363aa8640 10 monclient: build_initial_monmap 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.666+0000 7f0363aa8640 10 monclient: monmap: 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:34.666816+0000 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:34.666816+0000 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.667 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:34.668 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:34.668 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:34.668 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.666+0000 7f0363aa8640 1 Processor -- start 2024-03-25T18:13:34.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.666+0000 7f0363aa8640 1 -- start start 2024-03-25T18:13:34.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.666+0000 7f0363aa8640 10 monclient: init 2024-03-25T18:13:34.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:34.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:34.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f035c0b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:34.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f035c142a20 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:34.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f035c0a96b0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:34.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0a9d60 con 0x7f035c0a96b0 2024-03-25T18:13:34.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c13ff10 con 0x7f035c142a20 2024-03-25T18:13:34.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0aa1f0 con 0x7f035c0b4700 2024-03-25T18:13:34.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:34.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.667+0000 7f0363aa8640 10 monclient(hunting): authenticate will time out at 7257.997153s 2024-03-25T18:13:34.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.668+0000 7f0361043640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f035c142a20 0x7f035c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51396/0 (socket says 172.21.15.161:51396) 2024-03-25T18:13:34.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.668+0000 7f0361043640 1 -- 172.21.15.161:0/4068178002 learned_addr learned my addr 172.21.15.161:0/4068178002 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500948815 0 0) 0x7f035c13ff10 con 0x7f035c142a20 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0348003520 con 0x7f035c142a20 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347733096 0 0) 0x7f035c0a9d60 con 0x7f035c0a96b0 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:13:34.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c13ff10 con 0x7f035c0a96b0 2024-03-25T18:13:34.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373509405 0 0) 0x7f035c0aa1f0 con 0x7f035c0b4700 2024-03-25T18:13:34.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:13:34.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.670+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c0a9d60 con 0x7f035c0b4700 2024-03-25T18:13:34.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589581068 0 0) 0x7f0348003520 con 0x7f035c142a20 2024-03-25T18:13:34.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 >> v1:172.21.15.132:6790/0 conn(0x7f035c0b4700 legacy=0x7f035c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 >> v1:172.21.15.132:6789/0 conn(0x7f035c0a96b0 legacy=0x7f035c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 10 monclient: _finish_hunting 0 2024-03-25T18:13:34.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 1 monclient: found mon.noname-c 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c0b4ae0 con 0x7f035c142a20 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 10 monclient: _finish_auth 0 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 10 monclient: _check_auth_tickets 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0358001690 con 0x7f035c142a20 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.673+0000 7f0360842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.674+0000 7f0360842640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.674+0000 7f0360842640 10 monclient: dump: 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.674+0000 7f0363aa8640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:13:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.674+0000 7f0363aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03580017f0 con 0x7f035c142a20 2024-03-25T18:13:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0360842640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0363aa8640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0363aa8640 4 set_mon_vals no callback set 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0360842640 1 -- 172.21.15.161:0/4068178002 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0358001c00 con 0x7f035c142a20 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0360842640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0360842640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.675+0000 7f0360842640 10 monclient: dump: 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.680 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 10 monclient: shutdown 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 1 -- 172.21.15.161:0/4068178002 >> v1:172.21.15.161:6789/0 conn(0x7f035c142a20 legacy=0x7f035c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 1 -- 172.21.15.161:0/4068178002 shutdown_connections 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 1 -- 172.21.15.161:0/4068178002 >> 172.21.15.161:0/4068178002 conn(0x7f035c0668e0 msgr2=0x7f035c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 1 -- 172.21.15.161:0/4068178002 shutdown_connections 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.676+0000 7f0363aa8640 1 -- 172.21.15.161:0/4068178002 wait complete. 2024-03-25T18:13:34.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 10 monclient: build_initial_monmap 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 10 monclient: monmap: 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 1 librados: starting msgr at 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 1 librados: starting objecter 2024-03-25T18:13:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 1 Processor -- start 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 1 -- start start 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 1 librados: setting wanted keys 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 1 librados: calling monclient init 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.677+0000 7f0363aa8640 10 monclient: init 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f035c0a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f035c0b4700 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f035c142a20 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c1365c0 con 0x7f035c0b4700 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0ac120 con 0x7f035c0a96b0 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f035c0ac400 con 0x7f035c142a20 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0363aa8640 10 monclient(hunting): authenticate will time out at 7258.007825s 2024-03-25T18:13:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0361844640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f035c0a96b0 0x7f035c135b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51398/0 (socket says 172.21.15.161:51398) 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f0361844640 1 -- 172.21.15.161:0/3990607398 learned_addr learned my addr 172.21.15.161:0/3990607398 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f03527fc640 10 client.?.objecter ms_handle_connect 0x7f035c0a96b0 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f03527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f03527fc640 10 client.?.objecter ms_handle_connect 0x7f035c0b4700 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.678+0000 7f03527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.679+0000 7f03527fc640 10 client.?.objecter ms_handle_connect 0x7f035c142a20 2024-03-25T18:13:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.679+0000 7f03527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:34.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301254573 0 0) 0x7f035c1365c0 con 0x7f035c0b4700 2024-03-25T18:13:34.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:13:34.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0340004eb0 con 0x7f035c0b4700 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142546199 0 0) 0x7f035c0ac120 con 0x7f035c0a96b0 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.680+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c1365c0 con 0x7f035c0a96b0 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.681+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2801137041 0 0) 0x7f035c0ac400 con 0x7f035c142a20 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.681+0000 7f03527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.681+0000 7f03527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.681+0000 7f03527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.681+0000 7f03527fc640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:13:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.681+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f035c0ac120 con 0x7f035c142a20 2024-03-25T18:13:34.687 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.685+0000 7fd45a62d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3726014411 0 0) 0x7f035c1365c0 con 0x7f035c0a96b0 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f035c0ac400 con 0x7f035c0a96b0 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f034c001900 con 0x7f035c0a96b0 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): dump: 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.688 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 150835711 0 0) 0x7f0340004eb0 con 0x7f035c0b4700 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f035c1365c0 con 0x7f035c0b4700 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0358001bc0 con 0x7f035c0b4700 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.685+0000 7f03527fc640 10 monclient(hunting): dump: 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.689 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.686+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446458234 0 0) 0x7f035c0ac120 con 0x7f035c142a20 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.686+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0340004eb0 con 0x7f035c142a20 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.686+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0354001cb0 con 0x7f035c142a20 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.686+0000 7f03527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.686+0000 7f03527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.686+0000 7f03527fc640 10 monclient(hunting): dump: 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.690 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.691 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4122196257 0 0) 0x7f0340004eb0 con 0x7f035c142a20 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 >> v1:172.21.15.161:6789/0 conn(0x7f035c0a96b0 legacy=0x7f035c135b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 >> v1:172.21.15.132:6789/0 conn(0x7f035c0b4700 legacy=0x7f035c0ac820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 1 monclient: found mon.c 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f035c1350d0 con 0x7f035c142a20 2024-03-25T18:13:34.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f03527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 5 monclient: authenticate success, global_id 4133 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 10 monclient: _renew_subs 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035c136250 con 0x7f035c142a20 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 10 monclient: _renew_subs 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f035c133920 con 0x7f035c142a20 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.689+0000 7f0363aa8640 1 librados: init done 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.690+0000 7f0363aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.690+0000 7f0363aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.690+0000 7f0363aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:34.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.690+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f035c133700 con 0x7f035c142a20 2024-03-25T18:13:34.731 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.730+0000 7f2a02236040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:13:34.777 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.776+0000 7f2a02236040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03540020c0 con 0x7f035c142a20 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0354003b10 con 0x7f035c142a20 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 10 monclient: dump: 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:34.790 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0354004040 con 0x7f035c142a20 2024-03-25T18:13:34.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.789+0000 7f0353fff640 4 set_mon_vals no callback set 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 3390299863 0 0) 0x7f0354005660 con 0x7f035c142a20 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 10 client.4133.objecter ms_dispatch 0x7f035c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 3 client.4133.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 3 client.4133.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 20 client.4133.objecter dump_active .. 0 homeless 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0354005240 con 0x7f035c142a20 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.790+0000 7f03527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:34.826 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:34.825+0000 7fd45a62d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:13:34.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.862+0000 7f0363aa8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8", "id": 2}] 2024-03-25T18:13:34.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.862+0000 7f0363aa8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8", "id": 2}] 2024-03-25T18:13:34.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.862+0000 7f0363aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:34.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.862+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8", "id": 2} v 0) v1 -- 0x7f035c0b1d70 con 0x7f035c142a20 2024-03-25T18:13:34.891 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.890+0000 7f2a02236040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:13:34.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.905+0000 7f03527fc640 1 -- 172.21.15.161:0/3990607398 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3203813109 0 216599125) 0x7f03540054b0 con 0x7f035c142a20 2024-03-25T18:13:34.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.905+0000 7f03527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8", "id": 2}] 2024-03-25T18:13:34.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.905+0000 7f03527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:34.907 INFO:teuthology.orchestra.run.smithi161.stdout:2 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 10 librados: watch_flush enter 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 10 librados: watch_flush exit 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 20 client.4133.objecter shutdown clearing up homeless session... 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 10 client.4133.objecter successfully canceled tick 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 10 monclient: shutdown 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:34.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 >> v1:172.21.15.132:6790/0 conn(0x7f035c142a20 legacy=0x7f035c1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 shutdown_connections 2024-03-25T18:13:34.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 >> 172.21.15.161:0/3990607398 conn(0x7f035c0668e0 msgr2=0x7f035c13e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:34.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 shutdown_connections 2024-03-25T18:13:34.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 1 -- 172.21.15.161:0/3990607398 wait complete. 2024-03-25T18:13:34.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:34.906+0000 7f0363aa8640 1 librados: shutdown 2024-03-25T18:13:34.918 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 55459360-8371-4a31-91da-0b4dcb0523f0 3 2024-03-25T18:13:34.992 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:34.992+0000 7f2a02236040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:13:35.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.057+0000 7f9590e1e640 10 monclient: get_monmap_and_config 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.057+0000 7f9590e1e640 10 monclient: build_initial_monmap 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.058+0000 7f9590e1e640 10 monclient: monmap: 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:35.058717+0000 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:35.058717+0000 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.058+0000 7f9590e1e640 1 Processor -- start 2024-03-25T18:13:35.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.058+0000 7f9590e1e640 1 -- start start 2024-03-25T18:13:35.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.058+0000 7f9590e1e640 10 monclient: init 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f958c0b4700 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f958c142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f958c0a96b0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c0a9d60 con 0x7f958c142a20 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c13ff10 con 0x7f958c0b4700 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c0aa1f0 con 0x7f958c0a96b0 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.059+0000 7f9590e1e640 10 monclient(hunting): authenticate will time out at 7258.389118s 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.060+0000 7f958a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f958c0b4700 0x7f958c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51414/0 (socket says 172.21.15.161:51414) 2024-03-25T18:13:35.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.060+0000 7f958a59c640 1 -- 172.21.15.161:0/1027870350 learned_addr learned my addr 172.21.15.161:0/1027870350 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250949033 0 0) 0x7f958c13ff10 con 0x7f958c0b4700 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9574003520 con 0x7f958c0b4700 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964374727 0 0) 0x7f958c0a9d60 con 0x7f958c142a20 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c13ff10 con 0x7f958c142a20 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434707725 0 0) 0x7f958c0aa1f0 con 0x7f958c0a96b0 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:13:35.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.062+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c0a9d60 con 0x7f958c0a96b0 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727182659 0 0) 0x7f9574003520 con 0x7f958c0b4700 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 >> v1:172.21.15.132:6790/0 conn(0x7f958c0a96b0 legacy=0x7f958c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 >> v1:172.21.15.132:6789/0 conn(0x7f958c142a20 legacy=0x7f958c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: _finish_hunting 0 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 1 monclient: found mon.noname-c 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c0b4ae0 con 0x7f958c0b4700 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: _finish_auth 0 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: _check_auth_tickets 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f957c001690 con 0x7f958c0b4700 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f958959a640 10 monclient: dump: 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.067 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.068 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.068 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.068 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f9590e1e640 5 monclient: authenticate success, global_id 4135 2024-03-25T18:13:35.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.065+0000 7f9590e1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:35.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f957c0017f0 con 0x7f958c0b4700 2024-03-25T18:13:35.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f958959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:35.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f958959a640 1 -- 172.21.15.161:0/1027870350 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f957c001c00 con 0x7f958c0b4700 2024-03-25T18:13:35.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f9590e1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:35.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f9590e1e640 4 set_mon_vals no callback set 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f958959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f958959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f958959a640 10 monclient: dump: 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f9590e1e640 10 monclient: shutdown 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f9590e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.067+0000 7f9590e1e640 1 -- 172.21.15.161:0/1027870350 >> v1:172.21.15.161:6789/0 conn(0x7f958c0b4700 legacy=0x7f958c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.068+0000 7f9590e1e640 1 -- 172.21.15.161:0/1027870350 shutdown_connections 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.068+0000 7f9590e1e640 1 -- 172.21.15.161:0/1027870350 >> 172.21.15.161:0/1027870350 conn(0x7f958c0668e0 msgr2=0x7f958c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.068+0000 7f9590e1e640 1 -- 172.21.15.161:0/1027870350 shutdown_connections 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.068+0000 7f9590e1e640 1 -- 172.21.15.161:0/1027870350 wait complete. 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 10 monclient: build_initial_monmap 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 10 monclient: monmap: 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:35.073 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 1 librados: starting msgr at 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 1 librados: starting objecter 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 1 Processor -- start 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 1 -- start start 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 1 librados: setting wanted keys 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 1 librados: calling monclient init 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 10 monclient: init 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:35.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.069+0000 7f9590e1e640 10 monclient(hunting): picked mon.noname-a con 0x7f958c0a96b0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 10 monclient(hunting): picked mon.noname-c con 0x7f958c0b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 10 monclient(hunting): picked mon.noname-b con 0x7f958c142a20 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c0ae240 con 0x7f958c0a96b0 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c1d3fd0 con 0x7f958c142a20 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958c1d4160 con 0x7f958c0b4700 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f9590e1e640 10 monclient(hunting): authenticate will time out at 7258.399739s 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f958a59c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f958c0a96b0 0x7f958c0ac7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52658/0 (socket says 172.21.15.161:52658) 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f958a59c640 1 -- 172.21.15.161:0/124572975 learned_addr learned my addr 172.21.15.161:0/124572975 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f957affd640 10 client.?.objecter ms_handle_connect 0x7f958c0a96b0 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f957affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f957affd640 10 client.?.objecter ms_handle_connect 0x7f958c142a20 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f957affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f957affd640 10 client.?.objecter ms_handle_connect 0x7f958c0b4700 2024-03-25T18:13:35.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.070+0000 7f957affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:35.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566238971 0 0) 0x7f958c0ae240 con 0x7f958c0a96b0 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956c004eb0 con 0x7f958c0a96b0 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762606203 0 0) 0x7f958c1d3fd0 con 0x7f958c142a20 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c0ae240 con 0x7f958c142a20 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1429575220 0 0) 0x7f958c1d4160 con 0x7f958c0b4700 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:13:35.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.072+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c1d3fd0 con 0x7f958c0b4700 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3070661237 0 0) 0x7f956c004eb0 con 0x7f958c0a96b0 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f958c1d4160 con 0x7f958c0a96b0 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f957c001bc0 con 0x7f958c0a96b0 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): dump: 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.079 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 258183598 0 0) 0x7f958c1d3fd0 con 0x7f958c0b4700 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f956c004eb0 con 0x7f958c0b4700 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f95800018d0 con 0x7f958c0b4700 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): dump: 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.080 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2732714862 0 0) 0x7f958c0ae240 con 0x7f958c142a20 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f958c1d3fd0 con 0x7f958c142a20 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9584001bf0 con 0x7f958c142a20 2024-03-25T18:13:35.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.077+0000 7f957affd640 10 monclient(hunting): dump: 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.082 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684500323 0 0) 0x7f958c1d4160 con 0x7f958c0a96b0 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 >> v1:172.21.15.132:6790/0 conn(0x7f958c0b4700 legacy=0x7f958c1d4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 >> v1:172.21.15.161:6789/0 conn(0x7f958c142a20 legacy=0x7f958c1daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 1 monclient: found mon.a 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958c0acee0 con 0x7f958c0a96b0 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 10 monclient: _finish_auth 0 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:35.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f957affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 10 monclient: _renew_subs 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c1d4d40 con 0x7f958c0a96b0 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 10 monclient: _renew_subs 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c1db3e0 con 0x7f958c0a96b0 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.080+0000 7f9590e1e640 1 librados: init done 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.081+0000 7f9590e1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.081+0000 7f9590e1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.081+0000 7f9590e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:35.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.081+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c0ad130 con 0x7f958c0a96b0 2024-03-25T18:13:35.154 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:35.154+0000 7f2a02236040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:13:35.198 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:35.197+0000 7f2a02236040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:13:35.265 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.264+0000 7fd45a62d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:13:35.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f957c0011f0 con 0x7f958c0a96b0 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f957c004cb0 con 0x7f958c0a96b0 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 10 monclient: dump: 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.278 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f9588d99640 4 set_mon_vals no callback set 2024-03-25T18:13:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.276+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f957c005220 con 0x7f958c0a96b0 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 86574355 0 0) 0x7f957c006890 con 0x7f958c0a96b0 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 10 client.4137.objecter ms_dispatch 0x7f958c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f957c006b80 con 0x7f958c0a96b0 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.278+0000 7f957affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:35.312 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.311+0000 7fd45a62d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:13:35.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.350+0000 7f9590e1e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "55459360-8371-4a31-91da-0b4dcb0523f0", "id": 3}] 2024-03-25T18:13:35.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.350+0000 7f9590e1e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "55459360-8371-4a31-91da-0b4dcb0523f0", "id": 3}] 2024-03-25T18:13:35.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.350+0000 7f9590e1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:35.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.350+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd new", "uuid": "55459360-8371-4a31-91da-0b4dcb0523f0", "id": 3} v 0) v1 -- 0x7f958c04f730 con 0x7f958c0a96b0 2024-03-25T18:13:35.355 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:35.355+0000 7f2a02236040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:13:35.433 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.432+0000 7fd45a62d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:13:35.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.494+0000 7f957affd640 1 -- 172.21.15.161:0/124572975 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "55459360-8371-4a31-91da-0b4dcb0523f0", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 100348671 0 525046306) 0x7f957c006530 con 0x7f958c0a96b0 2024-03-25T18:13:35.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.494+0000 7f957affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "55459360-8371-4a31-91da-0b4dcb0523f0", "id": 3}] 2024-03-25T18:13:35.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.494+0000 7f957affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:35.495 INFO:teuthology.orchestra.run.smithi161.stdout:3 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 10 librados: watch_flush enter 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 10 librados: watch_flush exit 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 10 monclient: shutdown 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:35.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 >> v1:172.21.15.132:6789/0 conn(0x7f958c0a96b0 legacy=0x7f958c0ac7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 shutdown_connections 2024-03-25T18:13:35.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 >> 172.21.15.161:0/124572975 conn(0x7f958c0668e0 msgr2=0x7f958c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:35.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 shutdown_connections 2024-03-25T18:13:35.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 1 -- 172.21.15.161:0/124572975 wait complete. 2024-03-25T18:13:35.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.495+0000 7f9590e1e640 1 librados: shutdown 2024-03-25T18:13:35.508 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new dd421c87-126d-478b-b501-c1b526011ab9 4 2024-03-25T18:13:35.540 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.539+0000 7fd45a62d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:13:35.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.646+0000 7f8a154fb640 10 monclient: get_monmap_and_config 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.646+0000 7f8a154fb640 10 monclient: build_initial_monmap 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.646+0000 7f8a154fb640 10 monclient: monmap: 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:35.647273+0000 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:35.647273+0000 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:35.650 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:35.651 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 1 Processor -- start 2024-03-25T18:13:35.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 1 -- start start 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient: init 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8a1006c2d0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1006c6b0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8a10142d90 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a100b0840 con 0x7f8a1006c6b0 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a10143200 con 0x7f8a10142d90 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1006d810 con 0x7f8a1006c2d0 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.647+0000 7f8a154fb640 10 monclient(hunting): authenticate will time out at 7258.977459s 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.648+0000 7f8a0e7fc640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f8a1006c6b0 0x7f8a10142680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52672/0 (socket says 172.21.15.161:52672) 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.648+0000 7f8a0e7fc640 1 -- 172.21.15.161:0/2094995030 learned_addr learned my addr 172.21.15.161:0/2094995030 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:35.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887205796 0 0) 0x7f8a08001490 con 0x7f8a10142d90 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89f8003520 con 0x7f8a10142d90 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120050478 0 0) 0x7f8a100b0840 con 0x7f8a1006c6b0 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.650+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a08001490 con 0x7f8a1006c6b0 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.651+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860080949 0 0) 0x7f8a1006d810 con 0x7f8a1006c2d0 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.651+0000 7f8a0dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.651+0000 7f8a0dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.651+0000 7f8a0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.651+0000 7f8a0dffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:13:35.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.651+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a100b0840 con 0x7f8a1006c2d0 2024-03-25T18:13:35.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3008519224 0 0) 0x7f89f8003520 con 0x7f8a10142d90 2024-03-25T18:13:35.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 >> v1:172.21.15.132:6790/0 conn(0x7f8a1006c2d0 legacy=0x7f8a100afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 >> v1:172.21.15.132:6789/0 conn(0x7f8a1006c6b0 legacy=0x7f8a10142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 1 monclient: found mon.noname-c 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a100ae260 con 0x7f8a10142d90 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a08001a00 con 0x7f8a10142d90 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a154fb640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a154fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:35.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a0dffb640 10 monclient: dump: 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.653+0000 7f8a154fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a08001b60 con 0x7f8a10142d90 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a0dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a0dffb640 1 -- 172.21.15.161:0/2094995030 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a08001ef0 con 0x7f8a10142d90 2024-03-25T18:13:35.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a154fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a154fb640 4 set_mon_vals no callback set 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a0dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.655+0000 7f8a0dffb640 10 monclient: dump: 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 10 monclient: shutdown 2024-03-25T18:13:35.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 1 -- 172.21.15.161:0/2094995030 >> v1:172.21.15.161:6789/0 conn(0x7f8a10142d90 legacy=0x7f8a1014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 1 -- 172.21.15.161:0/2094995030 shutdown_connections 2024-03-25T18:13:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 1 -- 172.21.15.161:0/2094995030 >> 172.21.15.161:0/2094995030 conn(0x7f8a100b5e40 msgr2=0x7f8a100b6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 1 -- 172.21.15.161:0/2094995030 shutdown_connections 2024-03-25T18:13:35.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 1 -- 172.21.15.161:0/2094995030 wait complete. 2024-03-25T18:13:35.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 10 monclient: build_initial_monmap 2024-03-25T18:13:35.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.656+0000 7f8a154fb640 10 monclient: monmap: 2024-03-25T18:13:35.662 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:35.662 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 1 librados: starting msgr at 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 1 librados: starting objecter 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 1 Processor -- start 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 1 -- start start 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 1 librados: setting wanted keys 2024-03-25T18:13:35.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 1 librados: calling monclient init 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 10 monclient: init 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.657+0000 7f8a154fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 10 monclient(hunting): picked mon.noname-a con 0x7f8a1006c2d0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8a1006c6b0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8a10142d90 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1013dad0 con 0x7f8a1006c2d0 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1013d5b0 con 0x7f8a10142d90 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a1014e900 con 0x7f8a1006c6b0 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a154fb640 10 monclient(hunting): authenticate will time out at 7258.987786s 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a0f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8a10142d90 0x7f8a1013a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51434/0 (socket says 172.21.15.161:51434) 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a0effd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f8a1006c2d0 0x7f8a1013ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52684/0 (socket says 172.21.15.161:52684) 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f8a0f7fe640 1 -- 172.21.15.161:0/2424364435 learned_addr learned my addr 172.21.15.161:0/2424364435 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f89f77fe640 10 client.?.objecter ms_handle_connect 0x7f8a10142d90 2024-03-25T18:13:35.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.658+0000 7f89f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.659+0000 7f89f77fe640 10 client.?.objecter ms_handle_connect 0x7f8a1006c2d0 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.659+0000 7f89f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.659+0000 7f89f77fe640 10 client.?.objecter ms_handle_connect 0x7f8a1006c6b0 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.659+0000 7f89f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120605271 0 0) 0x7f8a1013dad0 con 0x7f8a1006c2d0 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc004eb0 con 0x7f8a1006c2d0 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314211573 0 0) 0x7f8a1013d5b0 con 0x7f8a10142d90 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.660+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a1013dad0 con 0x7f8a10142d90 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.661+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937562763 0 0) 0x7f8a1014e900 con 0x7f8a1006c6b0 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.661+0000 7f89f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.661+0000 7f89f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.661+0000 7f89f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.661+0000 7f89f77fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:13:35.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.661+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a1013d5b0 con 0x7f8a1006c6b0 2024-03-25T18:13:35.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2892728602 0 0) 0x7f8a1013dad0 con 0x7f8a10142d90 2024-03-25T18:13:35.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a1014e900 con 0x7f8a10142d90 2024-03-25T18:13:35.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a08003970 con 0x7f8a10142d90 2024-03-25T18:13:35.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): dump: 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086305768 0 0) 0x7f89fc004eb0 con 0x7f8a1006c2d0 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a1013dad0 con 0x7f8a1006c2d0 2024-03-25T18:13:35.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a000018d0 con 0x7f8a1006c2d0 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.665+0000 7f89f77fe640 10 monclient(hunting): dump: 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.666+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2628987337 0 0) 0x7f8a1013d5b0 con 0x7f8a1006c6b0 2024-03-25T18:13:35.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.666+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89fc004eb0 con 0x7f8a1006c6b0 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.666+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a04001930 con 0x7f8a1006c6b0 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.666+0000 7f89f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.666+0000 7f89f77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.666+0000 7f89f77fe640 10 monclient(hunting): dump: 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.670 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1239793849 0 0) 0x7f8a1014e900 con 0x7f8a10142d90 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 >> v1:172.21.15.132:6790/0 conn(0x7f8a1006c6b0 legacy=0x7f8a10139be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 >> v1:172.21.15.132:6789/0 conn(0x7f8a1006c2d0 legacy=0x7f8a1013ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 1 monclient: found mon.b 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1013d0d0 con 0x7f8a10142d90 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f89f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f8a154fb640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f8a154fb640 10 monclient: _renew_subs 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f8a154fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.668+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1013d2c0 con 0x7f8a10142d90 2024-03-25T18:13:35.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.669+0000 7f8a154fb640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.669+0000 7f8a154fb640 10 monclient: _renew_subs 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.669+0000 7f8a154fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.669+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1014e040 con 0x7f8a10142d90 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.669+0000 7f8a154fb640 1 librados: init done 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.669+0000 7f8a154fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.670+0000 7f8a154fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.670+0000 7f8a154fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.670+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a1004f730 con 0x7f8a10142d90 2024-03-25T18:13:35.697 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.697+0000 7fd45a62d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:13:35.740 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.739+0000 7fd45a62d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:13:35.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a08001640 con 0x7f8a10142d90 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a08003f20 con 0x7f8a10142d90 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 monclient: dump: 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:35.775 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8a080044d0 con 0x7f8a10142d90 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2151407291 0 0) 0x7f8a08006b50 con 0x7f8a10142d90 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f8a0d7fa640 4 set_mon_vals no callback set 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 client.4144.objecter ms_dispatch 0x7f8a10000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8a08018de0 con 0x7f8a10142d90 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:35.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.771+0000 7f89f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:35.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.844+0000 7f8a154fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dd421c87-126d-478b-b501-c1b526011ab9", "id": 4}] 2024-03-25T18:13:35.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.844+0000 7f8a154fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dd421c87-126d-478b-b501-c1b526011ab9", "id": 4}] 2024-03-25T18:13:35.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.844+0000 7f8a154fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:35.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.844+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dd421c87-126d-478b-b501-c1b526011ab9", "id": 4} v 0) v1 -- 0x7f8a10069f30 con 0x7f8a10142d90 2024-03-25T18:13:35.905 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:35.904+0000 7fd45a62d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:13:35.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.904+0000 7f89f77fe640 1 -- 172.21.15.161:0/2424364435 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dd421c87-126d-478b-b501-c1b526011ab9", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2837569208 0 1697012583) 0x7f8a08006710 con 0x7f8a10142d90 2024-03-25T18:13:35.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.904+0000 7f89f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dd421c87-126d-478b-b501-c1b526011ab9", "id": 4}] 2024-03-25T18:13:35.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.904+0000 7f89f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:35.905 INFO:teuthology.orchestra.run.smithi161.stdout:4 2024-03-25T18:13:35.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 10 librados: watch_flush enter 2024-03-25T18:13:35.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 10 librados: watch_flush exit 2024-03-25T18:13:35.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T18:13:35.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 10 client.4144.objecter successfully canceled tick 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 10 monclient: shutdown 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 >> v1:172.21.15.161:6789/0 conn(0x7f8a10142d90 legacy=0x7f8a1013a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 shutdown_connections 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 >> 172.21.15.161:0/2424364435 conn(0x7f8a100b5e40 msgr2=0x7f8a10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 shutdown_connections 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 1 -- 172.21.15.161:0/2424364435 wait complete. 2024-03-25T18:13:35.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:35.905+0000 7f8a154fb640 1 librados: shutdown 2024-03-25T18:13:35.916 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new d0a69e52-0ecc-4545-a092-69ba9d72c8be 5 2024-03-25T18:13:36.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.017+0000 7efda909a640 10 monclient: get_monmap_and_config 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.017+0000 7efda909a640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.017+0000 7efda909a640 10 monclient: monmap: 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:36.018124+0000 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:36.018124+0000 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:36.018 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 1 Processor -- start 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 1 -- start start 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient: init 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient(hunting): picked mon.noname-c con 0x7efda40b4700 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient(hunting): picked mon.noname-a con 0x7efda4142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient(hunting): picked mon.noname-b con 0x7efda40a96b0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40a9d60 con 0x7efda4142a20 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda413ff10 con 0x7efda40b4700 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40aa1f0 con 0x7efda40a96b0 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.018+0000 7efda909a640 10 monclient(hunting): authenticate will time out at 7259.348367s 2024-03-25T18:13:36.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.019+0000 7efda259c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7efda4142a20 0x7efda413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52692/0 (socket says 172.21.15.161:52692) 2024-03-25T18:13:36.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.019+0000 7efda259c640 1 -- 172.21.15.161:0/1512818937 learned_addr learned my addr 172.21.15.161:0/1512818937 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:36.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.021+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042569619 0 0) 0x7efda40a9d60 con 0x7efda4142a20 2024-03-25T18:13:36.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.021+0000 7efda1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.021+0000 7efda1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.021+0000 7efda1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.021+0000 7efda1d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.021+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c003520 con 0x7efda4142a20 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755536484 0 0) 0x7efda413ff10 con 0x7efda40b4700 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda40a9d60 con 0x7efda40b4700 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391023236 0 0) 0x7efda40aa1f0 con 0x7efda40a96b0 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:13:36.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.022+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda413ff10 con 0x7efda40a96b0 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237921387 0 0) 0x7efd8c003520 con 0x7efda4142a20 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 >> v1:172.21.15.132:6790/0 conn(0x7efda40a96b0 legacy=0x7efda413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 >> v1:172.21.15.161:6789/0 conn(0x7efda40b4700 legacy=0x7efda4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40b4ae0 con 0x7efda4142a20 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd98001690 con 0x7efda4142a20 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda1d9b640 10 monclient: dump: 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda909a640 5 monclient: authenticate success, global_id 4146 2024-03-25T18:13:36.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.025+0000 7efda909a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.027+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd980017f0 con 0x7efda4142a20 2024-03-25T18:13:36.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.027+0000 7efda1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.027+0000 7efda909a640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:36.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.027+0000 7efda909a640 4 set_mon_vals no callback set 2024-03-25T18:13:36.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda1d9b640 1 -- 172.21.15.161:0/1512818937 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd98001c00 con 0x7efda4142a20 2024-03-25T18:13:36.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda1d9b640 10 monclient: dump: 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 10 monclient: shutdown 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 1 -- 172.21.15.161:0/1512818937 >> v1:172.21.15.132:6789/0 conn(0x7efda4142a20 legacy=0x7efda413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 1 -- 172.21.15.161:0/1512818937 shutdown_connections 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 1 -- 172.21.15.161:0/1512818937 >> 172.21.15.161:0/1512818937 conn(0x7efda40668e0 msgr2=0x7efda4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 1 -- 172.21.15.161:0/1512818937 shutdown_connections 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.028+0000 7efda909a640 1 -- 172.21.15.161:0/1512818937 wait complete. 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 10 monclient: monmap: 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:36.031 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:36.032 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 1 librados: starting msgr at 2024-03-25T18:13:36.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 1 librados: starting objecter 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 1 Processor -- start 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 1 -- start start 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 1 librados: setting wanted keys 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 1 librados: calling monclient init 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.029+0000 7efda909a640 10 monclient: init 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient(hunting): picked mon.noname-b con 0x7efda40a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient(hunting): picked mon.noname-a con 0x7efda40b4700 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient(hunting): picked mon.noname-c con 0x7efda4142a20 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:36.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40abc60 con 0x7efda40b4700 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40ab880 con 0x7efda40a96b0 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40ac800 con 0x7efda4142a20 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda909a640 10 monclient(hunting): authenticate will time out at 7259.359998s 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda259c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7efda40b4700 0x7efda4136060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52694/0 (socket says 172.21.15.161:52694) 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.030+0000 7efda259c640 1 -- 172.21.15.161:0/3728371271 learned_addr learned my addr 172.21.15.161:0/3728371271 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.031+0000 7efd8b7fe640 10 client.?.objecter ms_handle_connect 0x7efda40b4700 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.031+0000 7efd8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.031+0000 7efd8b7fe640 10 client.?.objecter ms_handle_connect 0x7efda4142a20 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.031+0000 7efd8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.032+0000 7efd8b7fe640 10 client.?.objecter ms_handle_connect 0x7efda40a96b0 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.032+0000 7efd8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491000865 0 0) 0x7efda40abc60 con 0x7efda40b4700 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd90004eb0 con 0x7efda40b4700 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202010410 0 0) 0x7efda40ac800 con 0x7efda4142a20 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:13:36.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.033+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda40abc60 con 0x7efda4142a20 2024-03-25T18:13:36.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.034+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370836320 0 0) 0x7efda40ab880 con 0x7efda40a96b0 2024-03-25T18:13:36.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.034+0000 7efd8b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.034+0000 7efd8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.034+0000 7efd8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.034+0000 7efd8b7fe640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:13:36.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.034+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efda40ac800 con 0x7efda40a96b0 2024-03-25T18:13:36.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 119576420 0 0) 0x7efd90004eb0 con 0x7efda40b4700 2024-03-25T18:13:36.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda40ab880 con 0x7efda40b4700 2024-03-25T18:13:36.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd98001bc0 con 0x7efda40b4700 2024-03-25T18:13:36.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:36.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.038+0000 7efd8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3020661876 0 0) 0x7efda40ac800 con 0x7efda40a96b0 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd90004eb0 con 0x7efda40a96b0 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 204359954 0 0) 0x7efda40abc60 con 0x7efda4142a20 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efda40ac800 con 0x7efda4142a20 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd940018f0 con 0x7efda40a96b0 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.040 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd9c001bf0 con 0x7efda4142a20 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.039+0000 7efd8b7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.041 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.042 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.042 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.042 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.042 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.042 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3024211200 0 0) 0x7efda40ab880 con 0x7efda40b4700 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 >> v1:172.21.15.132:6790/0 conn(0x7efda4142a20 legacy=0x7efda41e2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 >> v1:172.21.15.161:6789/0 conn(0x7efda40a96b0 legacy=0x7efda40ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 1 monclient: found mon.a 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda41d7f80 con 0x7efda40b4700 2024-03-25T18:13:36.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efd8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 5 monclient: authenticate success, global_id 4149 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 10 monclient: _renew_subs 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda41d81d0 con 0x7efda40b4700 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 10 client.4149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 10 monclient: _renew_subs 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efda413bf00 con 0x7efda40b4700 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.041+0000 7efda909a640 1 librados: init done 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.042+0000 7efda909a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.042+0000 7efda909a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.042+0000 7efda909a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:36.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.042+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda40ad8d0 con 0x7efda40b4700 2024-03-25T18:13:36.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd980011f0 con 0x7efda40b4700 2024-03-25T18:13:36.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd98004c30 con 0x7efda40b4700 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 10 monclient: dump: 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efd98005120 con 0x7efda40b4700 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efda159a640 4 set_mon_vals no callback set 2024-03-25T18:13:36.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 344569951 0 0) 0x7efd98006880 con 0x7efda40b4700 2024-03-25T18:13:36.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 10 client.4149.objecter ms_dispatch 0x7efda4000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:13:36.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 3 client.4149.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:13:36.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 3 client.4149.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:13:36.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.046+0000 7efd8b7fe640 20 client.4149.objecter dump_active .. 0 homeless 2024-03-25T18:13:36.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.140+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efd98006410 con 0x7efda40b4700 2024-03-25T18:13:36.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.141+0000 7efd8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.141+0000 7efd8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:36.199 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:36.198+0000 7f2a02236040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:13:36.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.213+0000 7efda909a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d0a69e52-0ecc-4545-a092-69ba9d72c8be", "id": 5}] 2024-03-25T18:13:36.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.213+0000 7efda909a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d0a69e52-0ecc-4545-a092-69ba9d72c8be", "id": 5}] 2024-03-25T18:13:36.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.213+0000 7efda909a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:36.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.213+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d0a69e52-0ecc-4545-a092-69ba9d72c8be", "id": 5} v 0) v1 -- 0x7efda404f730 con 0x7efda40b4700 2024-03-25T18:13:36.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.297+0000 7efd8b7fe640 1 -- 172.21.15.161:0/3728371271 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0a69e52-0ecc-4545-a092-69ba9d72c8be", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3907940209 0 1988412176) 0x7efd98006a90 con 0x7efda40b4700 2024-03-25T18:13:36.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.297+0000 7efd8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d0a69e52-0ecc-4545-a092-69ba9d72c8be", "id": 5}] 2024-03-25T18:13:36.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.297+0000 7efd8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:36.298 INFO:teuthology.orchestra.run.smithi161.stdout:5 2024-03-25T18:13:36.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 10 librados: watch_flush enter 2024-03-25T18:13:36.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 10 librados: watch_flush exit 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 20 client.4149.objecter shutdown clearing up homeless session... 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 10 client.4149.objecter successfully canceled tick 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 10 monclient: shutdown 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 >> v1:172.21.15.132:6789/0 conn(0x7efda40b4700 legacy=0x7efda4136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 shutdown_connections 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.298+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 >> 172.21.15.161:0/3728371271 conn(0x7efda40668e0 msgr2=0x7efda40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.299+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 shutdown_connections 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.299+0000 7efda909a640 1 -- 172.21.15.161:0/3728371271 wait complete. 2024-03-25T18:13:36.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.299+0000 7efda909a640 1 librados: shutdown 2024-03-25T18:13:36.309 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new 4b872c23-c963-4cd7-89c0-336369dc9303 6 2024-03-25T18:13:36.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.409+0000 7f56ff932640 10 monclient: get_monmap_and_config 2024-03-25T18:13:36.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.409+0000 7f56ff932640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.409+0000 7f56ff932640 10 monclient: monmap: 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:36.409829+0000 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:36.409829+0000 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.409+0000 7f56ff932640 1 Processor -- start 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.409+0000 7f56ff932640 1 -- start start 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.409+0000 7f56ff932640 10 monclient: init 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient(hunting): picked mon.noname-a con 0x7f56f80a9920 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient(hunting): picked mon.noname-c con 0x7f56f80b45a0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient(hunting): picked mon.noname-b con 0x7f56f8146450 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f806b100 con 0x7f56f80a9920 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f80b2940 con 0x7f56f80b45a0 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f813fec0 con 0x7f56f8146450 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.410+0000 7f56ff932640 10 monclient(hunting): authenticate will time out at 7259.740185s 2024-03-25T18:13:36.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.411+0000 7f56fcecd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f56f80b45a0 0x7f56f8145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51462/0 (socket says 172.21.15.161:51462) 2024-03-25T18:13:36.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.411+0000 7f56fcecd640 1 -- 172.21.15.161:0/2166120900 learned_addr learned my addr 172.21.15.161:0/2166120900 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.413+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234781469 0 0) 0x7f56f80b2940 con 0x7f56f80b45a0 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.413+0000 7f56effff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.413+0000 7f56effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.413+0000 7f56effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.413+0000 7f56effff640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.413+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4003520 con 0x7f56f80b45a0 2024-03-25T18:13:36.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.414+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1546859465 0 0) 0x7f56f806b100 con 0x7f56f80a9920 2024-03-25T18:13:36.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.414+0000 7f56effff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.414+0000 7f56effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.414+0000 7f56effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.414+0000 7f56effff640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:13:36.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.414+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f80b2940 con 0x7f56f80a9920 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566873599 0 0) 0x7f56e4003520 con 0x7f56f80b45a0 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 >> v1:172.21.15.132:6790/0 conn(0x7f56f8146450 legacy=0x7f56f814dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 >> v1:172.21.15.132:6789/0 conn(0x7f56f80a9920 legacy=0x7f56f813f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 1 monclient: found mon.noname-c 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8140050 con 0x7f56f80b45a0 2024-03-25T18:13:36.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f56f4001690 con 0x7f56f80b45a0 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56ff932640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56ff932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56effff640 10 monclient: dump: 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.418 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.416+0000 7f56ff932640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f40017f0 con 0x7f56f80b45a0 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 4 set_mon_vals no callback set 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56effff640 1 -- 172.21.15.161:0/2166120900 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f56f4001a60 con 0x7f56f80b45a0 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56effff640 10 monclient: dump: 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.422 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 10 monclient: shutdown 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 1 -- 172.21.15.161:0/2166120900 >> v1:172.21.15.161:6789/0 conn(0x7f56f80b45a0 legacy=0x7f56f8145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 1 -- 172.21.15.161:0/2166120900 shutdown_connections 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 1 -- 172.21.15.161:0/2166120900 >> 172.21.15.161:0/2166120900 conn(0x7f56f80b7f40 msgr2=0x7f56f80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 1 -- 172.21.15.161:0/2166120900 shutdown_connections 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.418+0000 7f56ff932640 1 -- 172.21.15.161:0/2166120900 wait complete. 2024-03-25T18:13:36.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.419+0000 7f56ff932640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.419+0000 7f56ff932640 10 monclient: monmap: 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 librados: starting msgr at 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 librados: starting objecter 2024-03-25T18:13:36.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 Processor -- start 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 -- start start 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 librados: setting wanted keys 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 librados: calling monclient init 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient: init 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient(hunting): picked mon.noname-b con 0x7f56f80a9920 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient(hunting): picked mon.noname-a con 0x7f56f80b45a0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient(hunting): picked mon.noname-c con 0x7f56f8146450 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f8141c10 con 0x7f56f80b45a0 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f81425f0 con 0x7f56f80a9920 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f81d50b0 con 0x7f56f8146450 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.420+0000 7f56ff932640 10 monclient(hunting): authenticate will time out at 7259.750463s 2024-03-25T18:13:36.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.421+0000 7f56fdecf640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f56f8146450 0x7f56f81d49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.161:34200/0 (socket says 172.21.15.161:34200) 2024-03-25T18:13:36.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.421+0000 7f56fdecf640 1 -- 172.21.15.161:0/480640163 learned_addr learned my addr 172.21.15.161:0/480640163 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:36.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.421+0000 7f56edffb640 10 client.?.objecter ms_handle_connect 0x7f56f8146450 2024-03-25T18:13:36.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.421+0000 7f56edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.423+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060991055 0 0) 0x7f56f81d50b0 con 0x7f56f8146450 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.423+0000 7f56edffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.423+0000 7f56edffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.423+0000 7f56edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.423+0000 7f56edffb640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.423+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56dc004eb0 con 0x7f56f8146450 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.425+0000 7f56edffb640 10 client.?.objecter ms_handle_connect 0x7f56f80b45a0 2024-03-25T18:13:36.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.425+0000 7f56edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262791772 0 0) 0x7f56dc004eb0 con 0x7f56f8146450 2024-03-25T18:13:36.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56f81d50b0 con 0x7f56f8146450 2024-03-25T18:13:36.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f56f0001c90 con 0x7f56f8146450 2024-03-25T18:13:36.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 10 monclient(hunting): dump: 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:36.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 10 client.?.objecter ms_handle_connect 0x7f56f80a9920 2024-03-25T18:13:36.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.428+0000 7f56edffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4119470398 0 0) 0x7f56f81d50b0 con 0x7f56f8146450 2024-03-25T18:13:36.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 >> v1:172.21.15.161:6789/0 conn(0x7f56f80a9920 legacy=0x7f56f81daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 >> v1:172.21.15.132:6789/0 conn(0x7f56f80b45a0 legacy=0x7f56f8142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 1 monclient: found mon.c 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8141e60 con 0x7f56f8146450 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 5 monclient: authenticate success, global_id 4157 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 10 monclient: _renew_subs 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f8142140 con 0x7f56f8146450 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:36.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 10 monclient: _renew_subs 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f813c6b0 con 0x7f56f8146450 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.439+0000 7f56ff932640 1 librados: init done 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.440+0000 7f56ff932640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.440+0000 7f56ff932640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.440+0000 7f56ff932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:36.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.440+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f804f730 con 0x7f56f8146450 2024-03-25T18:13:36.495 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:36.494+0000 7f2a02236040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:13:36.547 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:36.547+0000 7f2a02236040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:13:36.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f0003990 con 0x7f56f8146450 2024-03-25T18:13:36.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f56f0003d60 con 0x7f56f8146450 2024-03-25T18:13:36.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 10 monclient: dump: 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f56f0004310 con 0x7f56f8146450 2024-03-25T18:13:36.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.573+0000 7f56ef7fe640 4 set_mon_vals no callback set 2024-03-25T18:13:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3842256978 0 0) 0x7f56f0006b40 con 0x7f56f8146450 2024-03-25T18:13:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 10 client.4157.objecter ms_dispatch 0x7f56f8000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:13:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 3 client.4157.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:13:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 3 client.4157.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:13:36.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 20 client.4157.objecter dump_active .. 0 homeless 2024-03-25T18:13:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f56f00047a0 con 0x7f56f8146450 2024-03-25T18:13:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:36.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.574+0000 7f56edffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:36.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.647+0000 7f56ff932640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4b872c23-c963-4cd7-89c0-336369dc9303", "id": 6}] 2024-03-25T18:13:36.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.647+0000 7f56ff932640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4b872c23-c963-4cd7-89c0-336369dc9303", "id": 6}] 2024-03-25T18:13:36.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.647+0000 7f56ff932640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:36.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.647+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd new", "uuid": "4b872c23-c963-4cd7-89c0-336369dc9303", "id": 6} v 0) v1 -- 0x7f56f80b6650 con 0x7f56f8146450 2024-03-25T18:13:36.667 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:36.667+0000 7f2a02236040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:13:36.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.679+0000 7f56edffb640 1 -- 172.21.15.161:0/480640163 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4b872c23-c963-4cd7-89c0-336369dc9303", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4130716287 0 1113811849) 0x7f56f00066a0 con 0x7f56f8146450 2024-03-25T18:13:36.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.679+0000 7f56edffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4b872c23-c963-4cd7-89c0-336369dc9303", "id": 6}] 2024-03-25T18:13:36.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.679+0000 7f56edffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:36.680 INFO:teuthology.orchestra.run.smithi161.stdout:6 2024-03-25T18:13:36.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 10 librados: watch_flush enter 2024-03-25T18:13:36.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 10 librados: watch_flush exit 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 20 client.4157.objecter shutdown clearing up homeless session... 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 10 client.4157.objecter successfully canceled tick 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 10 monclient: shutdown 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 >> v1:172.21.15.132:6790/0 conn(0x7f56f8146450 legacy=0x7f56f81d49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 shutdown_connections 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 >> 172.21.15.161:0/480640163 conn(0x7f56f80b7f40 msgr2=0x7f56f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.680+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 shutdown_connections 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.681+0000 7f56ff932640 1 -- 172.21.15.161:0/480640163 wait complete. 2024-03-25T18:13:36.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.681+0000 7f56ff932640 1 librados: shutdown 2024-03-25T18:13:36.694 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd new b0ef21de-3118-43fd-9712-52f4834fd5a1 7 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.792+0000 7f1151eb3640 10 monclient: get_monmap_and_config 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.792+0000 7f1151eb3640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.792+0000 7f1151eb3640 10 monclient: monmap: 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:13:36.793533+0000 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:13:36.793533+0000 2024-03-25T18:13:36.793 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.794 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.794 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:36.794 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:36.794 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:36.794 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.793+0000 7f1151eb3640 1 Processor -- start 2024-03-25T18:13:36.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.793+0000 7f1151eb3640 1 -- start start 2024-03-25T18:13:36.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.793+0000 7f1151eb3640 10 monclient: init 2024-03-25T18:13:36.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f114c0b4700 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f114c142a20 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f114c0a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0a9d60 con 0x7f114c142a20 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c13ff10 con 0x7f114c0a96b0 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0aa1f0 con 0x7f114c0b4700 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f1151eb3640 10 monclient(hunting): authenticate will time out at 7260.123905s 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f114affd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f114c142a20 0x7f114c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.161:52730/0 (socket says 172.21.15.161:52730) 2024-03-25T18:13:36.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.794+0000 7f114affd640 1 -- 172.21.15.161:0/940379382 learned_addr learned my addr 172.21.15.161:0/940379382 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:36.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.909+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785904961 0 0) 0x7f114c0aa1f0 con 0x7f114c0b4700 2024-03-25T18:13:36.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.909+0000 7f114a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.909+0000 7f114a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.909+0000 7f114a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.909+0000 7f114a7fc640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:13:36.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.909+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1134003520 con 0x7f114c0b4700 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1935763607 0 0) 0x7f1134003520 con 0x7f114c0b4700 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 >> v1:172.21.15.161:6789/0 conn(0x7f114c0a96b0 legacy=0x7f114c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 >> v1:172.21.15.132:6789/0 conn(0x7f114c142a20 legacy=0x7f114c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:36.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c0b4ae0 con 0x7f114c0b4700 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1140001690 con 0x7f114c0b4700 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f114a7fc640 10 monclient: dump: 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.915 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.916 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.916 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.916 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f1151eb3640 5 monclient: authenticate success, global_id 4160 2024-03-25T18:13:36.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.913+0000 7f1151eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11400017f0 con 0x7f114c0b4700 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f114a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f114a7fc640 1 -- 172.21.15.161:0/940379382 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1140001c40 con 0x7f114c0b4700 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f114a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f114a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f114a7fc640 10 monclient: dump: 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.921 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.922 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f1151eb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:36.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.920+0000 7f1151eb3640 4 set_mon_vals no callback set 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 10 monclient: shutdown 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 1 -- 172.21.15.161:0/940379382 >> v1:172.21.15.132:6790/0 conn(0x7f114c0b4700 legacy=0x7f114c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 1 -- 172.21.15.161:0/940379382 shutdown_connections 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 1 -- 172.21.15.161:0/940379382 >> 172.21.15.161:0/940379382 conn(0x7f114c0668e0 msgr2=0x7f114c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 1 -- 172.21.15.161:0/940379382 shutdown_connections 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 1 -- 172.21.15.161:0/940379382 wait complete. 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 10 monclient: build_initial_monmap 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.921+0000 7f1151eb3640 10 monclient: monmap: 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 0 2024-03-25T18:13:36.924 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 0.000000 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:created 0.000000 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 1 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 librados: starting msgr at 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 librados: starting objecter 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 Processor -- start 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 -- start start 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 librados: setting wanted keys 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 librados: calling monclient init 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient: init 2024-03-25T18:13:36.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f114c0a96b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f114c0b4700 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f114c142a20 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0ac140 con 0x7f114c0b4700 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0adde0 con 0x7f114c0a96b0 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0ae0c0 con 0x7f114c142a20 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:36.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.922+0000 7f1151eb3640 10 monclient(hunting): authenticate will time out at 7260.252674s 2024-03-25T18:13:36.926 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:36.922+0000 7fd45a62d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:13:36.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.988+0000 7f114b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f114c0a96b0 0x7f114c139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:51510/0 (socket says 172.21.15.161:51510) 2024-03-25T18:13:36.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.988+0000 7f114b7fe640 1 -- 172.21.15.161:0/2359670162 learned_addr learned my addr 172.21.15.161:0/2359670162 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-25T18:13:36.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.989+0000 7f1133fff640 10 client.?.objecter ms_handle_connect 0x7f114c0a96b0 2024-03-25T18:13:36.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.989+0000 7f1133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:36.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.991+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134931396 0 0) 0x7f114c0adde0 con 0x7f114c0a96b0 2024-03-25T18:13:36.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.991+0000 7f1133fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:36.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.991+0000 7f1133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:36.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.991+0000 7f1133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:36.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.991+0000 7f1133fff640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:13:36.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.991+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1138004eb0 con 0x7f114c0a96b0 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571629984 0 0) 0x7f1138004eb0 con 0x7f114c0a96b0 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f114c0adde0 con 0x7f114c0a96b0 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1140001690 con 0x7f114c0a96b0 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 10 monclient(hunting): dump: 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:36.997 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.996+0000 7f1133fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:36.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.998+0000 7f1133fff640 10 client.?.objecter ms_handle_connect 0x7f114c0b4700 2024-03-25T18:13:36.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:36.998+0000 7f1133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4051427079 0 0) 0x7f114c0adde0 con 0x7f114c0a96b0 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 >> v1:172.21.15.132:6790/0 conn(0x7f114c142a20 legacy=0x7f114c1e2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 >> v1:172.21.15.132:6789/0 conn(0x7f114c0b4700 legacy=0x7f114c13a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 1 monclient: found mon.b 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c0abe60 con 0x7f114c0a96b0 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:13:37.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 10 monclient: _renew_subs 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c0aba80 con 0x7f114c0a96b0 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 10 monclient: _renew_subs 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c0ac800 con 0x7f114c0a96b0 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.000+0000 7f1151eb3640 1 librados: init done 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.001+0000 7f1151eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:37.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.001+0000 7f1151eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:37.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.001+0000 7f1151eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:37.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.001+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c0abc70 con 0x7f114c0a96b0 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1140003960 con 0x7f114c0a96b0 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1140003c10 con 0x7f114c0a96b0 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 10 monclient: dump: 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:epoch 1 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.014 INFO:teuthology.orchestra.run.smithi161.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:election_strategy: 3 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1149ffb640 4 set_mon_vals no callback set 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.013+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2165429017 0 0) 0x7f1140003e30 con 0x7f114c0a96b0 2024-03-25T18:13:37.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.014+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1593012828 0 0) 0x7f1140015860 con 0x7f114c0a96b0 2024-03-25T18:13:37.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.014+0000 7f1133fff640 10 client.4159.objecter ms_dispatch 0x7f114c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:13:37.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.014+0000 7f1133fff640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:13:37.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.014+0000 7f1133fff640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:13:37.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.015+0000 7f1133fff640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-25T18:13:37.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.109+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1140014b20 con 0x7f114c0a96b0 2024-03-25T18:13:37.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.109+0000 7f1133fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:37.110 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.109+0000 7f1133fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:37.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.189+0000 7f1151eb3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b0ef21de-3118-43fd-9712-52f4834fd5a1", "id": 7}] 2024-03-25T18:13:37.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.189+0000 7f1151eb3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b0ef21de-3118-43fd-9712-52f4834fd5a1", "id": 7}] 2024-03-25T18:13:37.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.189+0000 7f1151eb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:37.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.189+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b0ef21de-3118-43fd-9712-52f4834fd5a1", "id": 7} v 0) v1 -- 0x7f114c04f730 con 0x7f114c0a96b0 2024-03-25T18:13:37.242 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:37.242+0000 7fd45a62d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:13:37.297 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:37.296+0000 7fd45a62d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:13:37.420 INFO:tasks.ceph.mgr.y.smithi132.stderr:2024-03-25T18:13:37.420+0000 7fd45a62d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:13:37.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.426+0000 7f1133fff640 1 -- 172.21.15.161:0/2359670162 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b0ef21de-3118-43fd-9712-52f4834fd5a1", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3863893404 0 1371665406) 0x7f1140015160 con 0x7f114c0a96b0 2024-03-25T18:13:37.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.426+0000 7f1133fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b0ef21de-3118-43fd-9712-52f4834fd5a1", "id": 7}] 2024-03-25T18:13:37.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.426+0000 7f1133fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:37.427 INFO:teuthology.orchestra.run.smithi161.stdout:7 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 10 librados: watch_flush enter 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 10 librados: watch_flush exit 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 10 client.4159.objecter successfully canceled tick 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 10 monclient: shutdown 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 >> v1:172.21.15.161:6789/0 conn(0x7f114c0a96b0 legacy=0x7f114c139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 shutdown_connections 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 >> 172.21.15.161:0/2359670162 conn(0x7f114c0668e0 msgr2=0x7f114c0b32e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:37.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 shutdown_connections 2024-03-25T18:13:37.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 1 -- 172.21.15.161:0/2359670162 wait complete. 2024-03-25T18:13:37.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-25T18:13:37.427+0000 7f1151eb3640 1 librados: shutdown 2024-03-25T18:13:37.438 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.438 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:13:37.438 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:13:37.440 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:13:37.440 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.440 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:13:37.440 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:13:37.442 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:13:37.442 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.443 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:13:37.443 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:13:37.445 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:13:37.445 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.445 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:13:37.445 DEBUG:teuthology.orchestra.run.smithi132:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:13:37.447 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:13:37.447 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.447 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:13:37.447 DEBUG:teuthology.orchestra.run.smithi161:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:13:37.450 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:13:37.450 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.450 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:13:37.450 DEBUG:teuthology.orchestra.run.smithi161:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:13:37.452 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:13:37.452 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.452 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:13:37.452 DEBUG:teuthology.orchestra.run.smithi161:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:13:37.454 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:13:37.454 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:13:37.455 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:13:37.455 DEBUG:teuthology.orchestra.run.smithi161:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:13:37.457 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:13:37.457 DEBUG:teuthology.orchestra.run.smithi132:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:37.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.609+0000 7fe320d23640 10 monclient: get_monmap_and_config 2024-03-25T18:13:37.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.609+0000 7fe320d23640 10 monclient: build_initial_monmap 2024-03-25T18:13:37.610 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.609+0000 7fe320d23640 10 monclient: monmap: 2024-03-25T18:13:37.610 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 0 2024-03-25T18:13:37.610 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:13:37.610513+0000 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:13:37.610513+0000 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:37.611 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.610+0000 7fe320d23640 1 Processor -- start 2024-03-25T18:13:37.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.610+0000 7fe320d23640 1 -- start start 2024-03-25T18:13:37.612 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.610+0000 7fe320d23640 10 monclient: init 2024-03-25T18:13:37.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:37.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:37.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient(hunting): picked mon.noname-b con 0x7fe31c0a9920 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:37.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient(hunting): picked mon.noname-c con 0x7fe31c0b45a0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:37.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient(hunting): picked mon.noname-a con 0x7fe31c146450 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:37.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c06b100 con 0x7fe31c146450 2024-03-25T18:13:37.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c0b2940 con 0x7fe31c0b45a0 2024-03-25T18:13:37.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c13fec0 con 0x7fe31c0a9920 2024-03-25T18:13:37.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:37.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.611+0000 7fe320d23640 10 monclient(hunting): authenticate will time out at 7257.874468s 2024-03-25T18:13:37.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.612+0000 7fe31a59c640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fe31c0a9920 0x7fe31c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:38302/0 (socket says 172.21.15.132:38302) 2024-03-25T18:13:37.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.612+0000 7fe31a59c640 1 -- 172.21.15.132:0/1811046890 learned_addr learned my addr 172.21.15.132:0/1811046890 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:37.638 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.638+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268210761 0 0) 0x7fe31c13fec0 con 0x7fe31c0a9920 2024-03-25T18:13:37.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.638+0000 7fe31959a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.638+0000 7fe31959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.638+0000 7fe31959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.638+0000 7fe31959a640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:13:37.639 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.638+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe304003520 con 0x7fe31c0a9920 2024-03-25T18:13:37.648 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.647+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863480330 0 0) 0x7fe304003520 con 0x7fe31c0a9920 2024-03-25T18:13:37.648 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.647+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 >> v1:172.21.15.161:6789/0 conn(0x7fe31c0b45a0 legacy=0x7fe31c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.647+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 >> v1:172.21.15.132:6789/0 conn(0x7fe31c146450 legacy=0x7fe31c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.647+0000 7fe31959a640 10 monclient: _finish_hunting 0 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.647+0000 7fe31959a640 1 monclient: found mon.noname-b 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.647+0000 7fe31959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c140050 con 0x7fe31c0a9920 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 10 monclient: _finish_auth 0 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 10 monclient: _check_auth_tickets 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe320d23640 5 monclient: authenticate success, global_id 4163 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe320d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe30c001690 con 0x7fe31c0a9920 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:37.649 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe31959a640 10 monclient: dump: 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:37.650 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.648+0000 7fe320d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe30c001920 con 0x7fe31c0a9920 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe31959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe31959a640 1 -- 172.21.15.132:0/1811046890 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe30c001b10 con 0x7fe31c0a9920 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe320d23640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe320d23640 4 set_mon_vals no callback set 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe31959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.755 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe31959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.754+0000 7fe31959a640 10 monclient: dump: 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:37.756 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 10 monclient: shutdown 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 1 -- 172.21.15.132:0/1811046890 >> v1:172.21.15.132:6790/0 conn(0x7fe31c0a9920 legacy=0x7fe31c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 1 -- 172.21.15.132:0/1811046890 shutdown_connections 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 1 -- 172.21.15.132:0/1811046890 >> 172.21.15.132:0/1811046890 conn(0x7fe31c0b7f40 msgr2=0x7fe31c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 1 -- 172.21.15.132:0/1811046890 shutdown_connections 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.755+0000 7fe320d23640 1 -- 172.21.15.132:0/1811046890 wait complete. 2024-03-25T18:13:37.758 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.756+0000 7fe320d23640 10 monclient: build_initial_monmap 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.756+0000 7fe320d23640 10 monclient: monmap: 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 0 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 0.000000 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:created 0.000000 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 1 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 1 librados: starting msgr at 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 1 librados: starting objecter 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 1 Processor -- start 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 1 -- start start 2024-03-25T18:13:37.759 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 1 librados: setting wanted keys 2024-03-25T18:13:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 1 librados: calling monclient init 2024-03-25T18:13:37.760 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.757+0000 7fe320d23640 10 monclient: init 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.758+0000 7fe320d23640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.758+0000 7fe320d23640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 10 monclient(hunting): picked mon.noname-a con 0x7fe31c0a9920 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe31a59c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe31c0a9920 0x7fe31c144830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:44902/0 (socket says 172.21.15.132:44902) 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe31a59c640 1 -- 172.21.15.132:0/2604032867 learned_addr learned my addr 172.21.15.132:0/2604032867 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 10 monclient(hunting): picked mon.noname-b con 0x7fe31c0b45a0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 10 monclient(hunting): picked mon.noname-c con 0x7fe31c146450 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c1415e0 con 0x7fe31c0a9920 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c0ac300 con 0x7fe31c0b45a0 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe31c0ac5e0 con 0x7fe31c146450 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:37.761 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe320d23640 10 monclient(hunting): authenticate will time out at 7258.022462s 2024-03-25T18:13:37.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe30affd640 10 client.?.objecter ms_handle_connect 0x7fe31c0a9920 2024-03-25T18:13:37.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.759+0000 7fe30affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.760+0000 7fe30affd640 10 client.?.objecter ms_handle_connect 0x7fe31c0b45a0 2024-03-25T18:13:37.762 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.760+0000 7fe30affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.772+0000 7fe30affd640 10 client.?.objecter ms_handle_connect 0x7fe31c146450 2024-03-25T18:13:37.773 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.772+0000 7fe30affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.910+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754966396 0 0) 0x7fe31c1415e0 con 0x7fe31c0a9920 2024-03-25T18:13:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.910+0000 7fe30affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.911 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.910+0000 7fe30affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.910+0000 7fe30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.910+0000 7fe30affd640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:13:37.912 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.910+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc004eb0 con 0x7fe31c0a9920 2024-03-25T18:13:37.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.918+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932275377 0 0) 0x7fe2fc004eb0 con 0x7fe31c0a9920 2024-03-25T18:13:37.919 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.918+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe31c1415e0 con 0x7fe31c0a9920 2024-03-25T18:13:37.922 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe30c0047e0 con 0x7fe31c0a9920 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): dump: 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:37.923 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586453962 0 0) 0x7fe31c0ac300 con 0x7fe31c0b45a0 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 10 monclient(hunting): my global_id is 4171 2024-03-25T18:13:37.924 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.922+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2fc004eb0 con 0x7fe31c0b45a0 2024-03-25T18:13:37.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.924+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337835991 0 0) 0x7fe31c0ac5e0 con 0x7fe31c146450 2024-03-25T18:13:37.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.924+0000 7fe30affd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:37.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.924+0000 7fe30affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:37.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.924+0000 7fe30affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:37.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.924+0000 7fe30affd640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:13:37.925 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.924+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe31c0ac300 con 0x7fe31c146450 2024-03-25T18:13:37.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.993+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1834795738 0 0) 0x7fe31c0ac300 con 0x7fe31c146450 2024-03-25T18:13:37.994 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:37.993+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe31c0ac5e0 con 0x7fe31c146450 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.052+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe314001ad0 con 0x7fe31c146450 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.052+0000 7fe30affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.052+0000 7fe30affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.052+0000 7fe30affd640 10 monclient(hunting): dump: 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:38.053 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:38.054 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:38.054 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3950145505 0 0) 0x7fe31c0ac5e0 con 0x7fe31c146450 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 >> v1:172.21.15.161:6789/0 conn(0x7fe31c0b45a0 legacy=0x7fe31c1d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 >> v1:172.21.15.132:6789/0 conn(0x7fe31c0a9920 legacy=0x7fe31c144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 1 monclient: found mon.c 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c144f40 con 0x7fe31c146450 2024-03-25T18:13:38.095 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 10 monclient: _finish_auth 0 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe30affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe320d23640 5 monclient: authenticate success, global_id 4166 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe320d23640 10 monclient: _renew_subs 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe320d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.093+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31c145190 con 0x7fe31c146450 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.094+0000 7fe320d23640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.094+0000 7fe320d23640 10 monclient: _renew_subs 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.094+0000 7fe320d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.094+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31c1d7f10 con 0x7fe31c146450 2024-03-25T18:13:38.096 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.094+0000 7fe320d23640 1 librados: init done 2024-03-25T18:13:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.095+0000 7fe320d23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.095+0000 7fe320d23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.095+0000 7fe320d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:38.097 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.095+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c04f730 con 0x7fe31c146450 2024-03-25T18:13:38.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe314001fa0 con 0x7fe31c146450 2024-03-25T18:13:38.107 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe30affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe314003b10 con 0x7fe31c146450 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe30affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe30affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe30affd640 10 monclient: dump: 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:epoch 1 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:13:38.108 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.107+0000 7fe318d99640 4 set_mon_vals no callback set 2024-03-25T18:13:38.158 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.158+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 2165429017 0 0) 0x7fe314003c70 con 0x7fe31c146450 2024-03-25T18:13:38.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.160+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fe314014680 con 0x7fe31c146450 2024-03-25T18:13:38.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.160+0000 7fe30affd640 10 client.4166.objecter ms_dispatch 0x7fe31c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:38.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.160+0000 7fe30affd640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:38.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.160+0000 7fe30affd640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:38.161 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.160+0000 7fe30affd640 20 client.4166.objecter dump_active .. 0 homeless 2024-03-25T18:13:38.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.288+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe314013d50 con 0x7fe31c146450 2024-03-25T18:13:38.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.288+0000 7fe30affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:38.289 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.288+0000 7fe30affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:38.413 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.412+0000 7fe320d23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:38.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.412+0000 7fe320d23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:38.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.412+0000 7fe320d23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:13:38.414 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.412+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe31c13b4d0 con 0x7fe31c146450 2024-03-25T18:13:38.613 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.612+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3582916369 0 0) 0x7fe314004390 con 0x7fe31c146450 2024-03-25T18:13:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.613+0000 7fe31a59c640 10 monclient: get_auth_request con 0x7fe2fc06e1f0 auth_method 0 2024-03-25T18:13:38.614 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.613+0000 7fe30affd640 10 client.4166.objecter ms_handle_connect 0x7fe2fc06e1f0 2024-03-25T18:13:38.683 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:38.683+0000 7f29a3010640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:13:38.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.696+0000 7fe30affd640 1 -- 172.21.15.132:0/2604032867 <== mon.2 v1:172.21.15.132:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fe314008e50 con 0x7fe31c146450 2024-03-25T18:13:38.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.696+0000 7fe30affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:38.697 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.696+0000 7fe30affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:38.697 INFO:teuthology.orchestra.run.smithi132.stdout: 2024-03-25T18:13:38.697 INFO:teuthology.orchestra.run.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:38.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.697+0000 7fe320d23640 10 librados: watch_flush enter 2024-03-25T18:13:38.699 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 10 librados: watch_flush exit 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 20 client.4166.objecter shutdown clearing up homeless session... 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 10 client.4166.objecter successfully canceled tick 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 >> v1:172.21.15.161:6800/411950978 conn(0x7fe2fc06e1f0 legacy=0x7fe2fc0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 10 monclient: shutdown 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 >> v1:172.21.15.132:6790/0 conn(0x7fe31c146450 legacy=0x7fe31c1e1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 shutdown_connections 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 >> 172.21.15.132:0/2604032867 conn(0x7fe31c0b7f40 msgr2=0x7fe31c13f580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 shutdown_connections 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 -- 172.21.15.132:0/2604032867 wait complete. 2024-03-25T18:13:38.700 INFO:teuthology.orchestra.run.smithi132.stderr:2024-03-25T18:13:38.698+0000 7fe320d23640 1 librados: shutdown 2024-03-25T18:13:38.714 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:13:38.715 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:13:38.836 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:13:38.835+0000 7f29a3010640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:13:39.520 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:39.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.668+0000 7f3843293640 10 monclient: get_monmap_and_config 2024-03-25T18:13:39.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.668+0000 7f3843293640 10 monclient: build_initial_monmap 2024-03-25T18:13:39.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.668+0000 7f3843293640 10 monclient: monmap: 2024-03-25T18:13:39.669 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:13:39.669 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:39.669 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:13:39.669643+0000 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:13:39.669643+0000 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:39.670 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.669+0000 7f3843293640 1 Processor -- start 2024-03-25T18:13:39.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.669+0000 7f3843293640 1 -- start start 2024-03-25T18:13:39.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.669+0000 7f3843293640 10 monclient: init 2024-03-25T18:13:39.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:39.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:39.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient(hunting): picked mon.noname-c con 0x7f383c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient(hunting): picked mon.noname-a con 0x7f383c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient(hunting): picked mon.noname-b con 0x7f383c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c142520 con 0x7f383c13f710 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c0b28a0 con 0x7f383c0a98f0 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c0b2a30 con 0x7f383c1430c0 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.670+0000 7f3843293640 10 monclient(hunting): authenticate will time out at 7259.933526s 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.671+0000 7f3841830640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f383c1430c0 0x7f383c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:38322/0 (socket says 172.21.15.132:38322) 2024-03-25T18:13:39.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.671+0000 7f3841830640 1 -- 172.21.15.132:0/2159416854 learned_addr learned my addr 172.21.15.132:0/2159416854 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230399898 0 0) 0x7f383c142520 con 0x7f383c13f710 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003520 con 0x7f383c13f710 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3930411066 0 0) 0x7f383c0b28a0 con 0x7f383c0a98f0 2024-03-25T18:13:39.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 10 monclient(hunting): my global_id is 4174 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.675+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c142520 con 0x7f383c0a98f0 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.676+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483028490 0 0) 0x7f383c0b2a30 con 0x7f383c1430c0 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.676+0000 7f3833fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.676+0000 7f3833fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.676+0000 7f3833fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.676+0000 7f3833fff640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:13:39.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.676+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c0b28a0 con 0x7f383c1430c0 2024-03-25T18:13:39.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1328410675 0 0) 0x7f3828003520 con 0x7f383c13f710 2024-03-25T18:13:39.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 >> v1:172.21.15.132:6790/0 conn(0x7f383c1430c0 legacy=0x7f383c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 >> v1:172.21.15.161:6789/0 conn(0x7f383c0a98f0 legacy=0x7f383c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:39.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 1 monclient: found mon.noname-a 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c0b44c0 con 0x7f383c13f710 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f382c0015c0 con 0x7f383c13f710 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3833fff640 10 monclient: dump: 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.683 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:39.684 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:39.684 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:39.684 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:39.684 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3843293640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:13:39.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.681+0000 7f3843293640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:39.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.684+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f382c001720 con 0x7f383c13f710 2024-03-25T18:13:39.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.684+0000 7f3833fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:39.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.684+0000 7f3833fff640 1 -- 172.21.15.132:0/2159416854 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f382c001b30 con 0x7f383c13f710 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.684+0000 7f3833fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.684+0000 7f3833fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.685+0000 7f3833fff640 10 monclient: dump: 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.685+0000 7f3843293640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.685+0000 7f3843293640 4 set_mon_vals no callback set 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 10 monclient: shutdown 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:39.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 1 -- 172.21.15.132:0/2159416854 >> v1:172.21.15.132:6789/0 conn(0x7f383c13f710 legacy=0x7f383c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 1 -- 172.21.15.132:0/2159416854 shutdown_connections 2024-03-25T18:13:39.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 1 -- 172.21.15.132:0/2159416854 >> 172.21.15.132:0/2159416854 conn(0x7f383c0b7ea0 msgr2=0x7f383c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:39.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 1 -- 172.21.15.132:0/2159416854 shutdown_connections 2024-03-25T18:13:39.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.686+0000 7f3843293640 1 -- 172.21.15.132:0/2159416854 wait complete. 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.687+0000 7f3843293640 10 monclient: build_initial_monmap 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.687+0000 7f3843293640 10 monclient: monmap: 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:39.690 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.687+0000 7f3843293640 1 librados: starting msgr at 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.687+0000 7f3843293640 1 librados: starting objecter 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.687+0000 7f3843293640 1 Processor -- start 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 1 -- start start 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 1 librados: setting wanted keys 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 1 librados: calling monclient init 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 10 monclient: init 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 10 monclient(hunting): picked mon.noname-c con 0x7f383c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 10 monclient(hunting): picked mon.noname-b con 0x7f383c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 10 monclient(hunting): picked mon.noname-a con 0x7f383c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.688+0000 7f3843293640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c13c000 con 0x7f383c1430c0 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.689+0000 7f3843293640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c06e9a0 con 0x7f383c13f710 2024-03-25T18:13:39.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.689+0000 7f3843293640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f383c06eb30 con 0x7f383c0a98f0 2024-03-25T18:13:39.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.689+0000 7f3843293640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:39.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.689+0000 7f3843293640 10 monclient(hunting): authenticate will time out at 7259.951787s 2024-03-25T18:13:39.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.689+0000 7f3841830640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f383c1430c0 0x7f383c070eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:44914/0 (socket says 172.21.15.132:44914) 2024-03-25T18:13:39.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.689+0000 7f3841830640 1 -- 172.21.15.132:0/2547824193 learned_addr learned my addr 172.21.15.132:0/2547824193 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:39.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.690+0000 7f3831ffb640 10 client.?.objecter ms_handle_connect 0x7f383c1430c0 2024-03-25T18:13:39.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.690+0000 7f3831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:39.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.690+0000 7f3831ffb640 10 client.?.objecter ms_handle_connect 0x7f383c13f710 2024-03-25T18:13:39.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.690+0000 7f3831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:39.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.690+0000 7f3831ffb640 10 client.?.objecter ms_handle_connect 0x7f383c0a98f0 2024-03-25T18:13:39.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.690+0000 7f3831ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425345172 0 0) 0x7f383c13c000 con 0x7f383c1430c0 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): my global_id is 4176 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3820004eb0 con 0x7f383c1430c0 2024-03-25T18:13:39.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637442591 0 0) 0x7f383c06e9a0 con 0x7f383c13f710 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 10 monclient(hunting): my global_id is 4177 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.693+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c13c000 con 0x7f383c13f710 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.694+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576152359 0 0) 0x7f383c06eb30 con 0x7f383c0a98f0 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.694+0000 7f3831ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.694+0000 7f3831ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.694+0000 7f3831ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.694+0000 7f3831ffb640 10 monclient(hunting): my global_id is 4172 2024-03-25T18:13:39.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.694+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c06e9a0 con 0x7f383c0a98f0 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2987750157 0 0) 0x7f383c13c000 con 0x7f383c13f710 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f383c06eb30 con 0x7f383c13f710 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523026435 0 0) 0x7f3820004eb0 con 0x7f383c1430c0 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f383c13c000 con 0x7f383c1430c0 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f382c001a30 con 0x7f383c13f710 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.703 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3834001b50 con 0x7f383c1430c0 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.701+0000 7f3831ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.704 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:39.705 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:39.705 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:39.705 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:39.705 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.704+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4042619468 0 0) 0x7f383c06e9a0 con 0x7f383c0a98f0 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.704+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3820004eb0 con 0x7f383c0a98f0 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.704+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f38380018f0 con 0x7f383c0a98f0 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.704+0000 7f3831ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.704+0000 7f3831ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.704+0000 7f3831ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:39.706 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:39.707 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:39.707 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2244934332 0 0) 0x7f383c13c000 con 0x7f383c1430c0 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 >> v1:172.21.15.132:6790/0 conn(0x7f383c0a98f0 legacy=0x7f383c06e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 >> v1:172.21.15.161:6789/0 conn(0x7f383c13f710 legacy=0x7f383c138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 1 monclient: found mon.a 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:39.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c06ecc0 con 0x7f383c1430c0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3831ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3843293640 5 monclient: authenticate success, global_id 4176 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3843293640 10 monclient: _renew_subs 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3843293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.708+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c139670 con 0x7f383c1430c0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.709+0000 7f3843293640 10 client.4176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.709+0000 7f3843293640 10 monclient: _renew_subs 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.709+0000 7f3843293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.709+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c0715c0 con 0x7f383c1430c0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.709+0000 7f3843293640 1 librados: init done 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.710+0000 7f3843293640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.710+0000 7f3843293640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.710+0000 7f3843293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:39.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.710+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383c070170 con 0x7f383c1430c0 2024-03-25T18:13:39.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.712+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3834002000 con 0x7f383c1430c0 2024-03-25T18:13:39.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.712+0000 7f3831ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:39.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:39.712+0000 7f38337fe640 4 set_mon_vals no callback set 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.130+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3834004b60 con 0x7f383c1430c0 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.130+0000 7f3831ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.130+0000 7f3831ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.130+0000 7f3831ffb640 10 monclient: dump: 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:40.131 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:40.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.132+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f3834004cc0 con 0x7f383c1430c0 2024-03-25T18:13:40.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.132+0000 7f384102f640 10 monclient: get_auth_request con 0x7f382006e0b0 auth_method 0 2024-03-25T18:13:40.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.133+0000 7f3831ffb640 10 client.4176.objecter ms_handle_connect 0x7f382006e0b0 2024-03-25T18:13:40.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f3834054950 con 0x7f383c1430c0 2024-03-25T18:13:40.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 10 client.4176.objecter ms_dispatch 0x7f383c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:40.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 3 client.4176.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:40.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 3 client.4176.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:40.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 20 client.4176.objecter dump_active .. 0 homeless 2024-03-25T18:13:40.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3834050290 con 0x7f383c1430c0 2024-03-25T18:13:40.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:40.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.143+0000 7f3831ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:40.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.505+0000 7f3843293640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:40.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.505+0000 7f3843293640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:40.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.505+0000 7f3843293640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:40.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.505+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f383c070550 con 0x7f383c1430c0 2024-03-25T18:13:40.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.575+0000 7f3831ffb640 1 -- 172.21.15.132:0/2547824193 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f383c070550 con 0x7f383c1430c0 2024-03-25T18:13:40.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.575+0000 7f3831ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:40.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.575+0000 7f3831ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:40.576 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:13:40.576 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.577+0000 7f3843293640 10 librados: watch_flush enter 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 10 librados: watch_flush exit 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 20 client.4176.objecter shutdown clearing up homeless session... 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 10 client.4176.objecter successfully canceled tick 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 >> v1:172.21.15.161:6800/411950978 conn(0x7f382006e0b0 legacy=0x7f3820070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 10 monclient: shutdown 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 >> v1:172.21.15.132:6789/0 conn(0x7f383c1430c0 legacy=0x7f383c070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 shutdown_connections 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 >> 172.21.15.132:0/2547824193 conn(0x7f383c0b7ea0 msgr2=0x7f383c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:40.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.578+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 shutdown_connections 2024-03-25T18:13:40.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.579+0000 7f3843293640 1 -- 172.21.15.132:0/2547824193 wait complete. 2024-03-25T18:13:40.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:40.579+0000 7f3843293640 1 librados: shutdown 2024-03-25T18:13:40.594 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:44.219 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:06.698 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.219 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:06.699 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.284 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:06.763 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.284 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:06.763 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.345 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:06.826 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.404 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:06.885 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.508 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:06.994 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.565 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:07.051 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.641 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:07.124 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.667 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:07.154 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.703 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:07.186 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.704 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:07.172 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.727 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:07.214 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.779 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:07.246 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:44.830 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-03-25T18:13:44.824+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:44.833 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-03-25T18:13:44.827+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:44.969 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-03-25T18:13:44.962+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:45.000 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:07.467 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.061 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:07.527 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.136 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-03-25T18:13:45.129+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:45.276 INFO:tasks.ceph.osd.2.smithi132.stderr:2024-03-25T18:13:45.267+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:45.276 INFO:tasks.ceph.osd.0.smithi132.stderr:2024-03-25T18:13:45.270+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:45.292 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:07.771 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.314 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:07.793 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.329 INFO:tasks.ceph.osd.3.smithi132.stderr:2024-03-25T18:13:45.322+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:45.429 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:07.910 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.593 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:08.079 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.661 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:08.140 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.662 INFO:tasks.ceph.osd.1.smithi132.stderr:2024-03-25T18:13:45.654+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:13:45.691 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:08.170 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.742 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:08.225 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.744 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:08.231 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.799 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:08.280 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.845 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:08.312 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.956 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:08.442 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:45.978 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:08.457 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.006 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:08.485 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.114 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:08.595 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.129 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:08.596 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.137 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:08.620 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.147 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:08.633 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.240 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:08.707 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.246 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:08.725 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.282 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:08.761 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.322 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:08.808 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.379 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:08.860 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.464 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:08.947 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.496 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:08.963 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.504 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:08.991 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.514 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:08.993 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.551 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:09.030 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.592 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:09.059 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.609 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:09.095 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.645 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:09.127 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.730 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:09.213 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.773 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:09.260 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.778 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:09.257 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.820 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:09.299 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.851 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:09.318 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.883 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:09.351 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.896 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:09.382 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.922 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:09.403 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:46.994 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:09.477 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.041 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:09.527 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.044 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:09.523 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.086 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:09.565 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.141 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:09.608 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.171 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:09.639 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.206 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:09.687 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.206 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:09.692 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.260 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:09.743 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.309 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:09.788 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.310 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:09.797 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.351 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:09.830 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.399 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:47.436 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:09.903 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.469 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:09.936 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.475 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:09.956 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.475 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:09.961 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.496+0000 7fa10b14b640 10 monclient: get_monmap_and_config 2024-03-25T18:13:47.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.496+0000 7fa10b14b640 10 monclient: build_initial_monmap 2024-03-25T18:13:47.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.496+0000 7fa10b14b640 10 monclient: monmap: 2024-03-25T18:13:47.496 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:13:47.497108+0000 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:13:47.497108+0000 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:47.497 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.496+0000 7fa10b14b640 1 Processor -- start 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.496+0000 7fa10b14b640 1 -- start start 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.496+0000 7fa10b14b640 10 monclient: init 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient(hunting): picked mon.noname-c con 0x7fa1040b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient(hunting): picked mon.noname-a con 0x7fa1040b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient(hunting): picked mon.noname-b con 0x7fa10414be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040af940 con 0x7fa1040b1730 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040b0a70 con 0x7fa1040b4d60 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040b0d50 con 0x7fa10414be90 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:47.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa10b14b640 10 monclient(hunting): authenticate will time out at 7267.760162s 2024-03-25T18:13:47.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa103fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa1040b1730 0x7fa1040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:48916/0 (socket says 172.21.15.132:48916) 2024-03-25T18:13:47.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.497+0000 7fa103fff640 1 -- 172.21.15.132:0/3037369808 learned_addr learned my addr 172.21.15.132:0/3037369808 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666309116 0 0) 0x7fa1040b0a70 con 0x7fa1040b4d60 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f4003520 con 0x7fa1040b4d60 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682483058 0 0) 0x7fa1040af940 con 0x7fa1040b1730 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:13:47.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.500+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1040b0a70 con 0x7fa1040b1730 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 942587477 0 0) 0x7fa0f4003520 con 0x7fa1040b4d60 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 >> v1:172.21.15.132:6790/0 conn(0x7fa10414be90 legacy=0x7fa10414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 >> v1:172.21.15.132:6789/0 conn(0x7fa1040b1730 legacy=0x7fa1040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 1 monclient: found mon.noname-c 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:47.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1040b1030 con 0x7fa1040b4d60 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa0f8001690 con 0x7fa1040b4d60 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa1037fe640 10 monclient: dump: 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.505 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:47.506 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:47.506 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:47.506 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa10b14b640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:13:47.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.503+0000 7fa10b14b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:47.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f80017f0 con 0x7fa1040b4d60 2024-03-25T18:13:47.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa1037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:47.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa1037fe640 1 -- 172.21.15.132:0/3037369808 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa0f8001c00 con 0x7fa1040b4d60 2024-03-25T18:13:47.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa1037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa1037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:47.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa1037fe640 10 monclient: dump: 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa10b14b640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.505+0000 7fa10b14b640 4 set_mon_vals no callback set 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 10 monclient: shutdown 2024-03-25T18:13:47.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 1 -- 172.21.15.132:0/3037369808 >> v1:172.21.15.161:6789/0 conn(0x7fa1040b4d60 legacy=0x7fa10413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 1 -- 172.21.15.132:0/3037369808 shutdown_connections 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 1 -- 172.21.15.132:0/3037369808 >> 172.21.15.132:0/3037369808 conn(0x7fa1040617a0 msgr2=0x7fa104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 1 -- 172.21.15.132:0/3037369808 shutdown_connections 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.506+0000 7fa10b14b640 1 -- 172.21.15.132:0/3037369808 wait complete. 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 10 monclient: build_initial_monmap 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 10 monclient: monmap: 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:47.509 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:47.510 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 1 librados: starting msgr at 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 1 librados: starting objecter 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 1 Processor -- start 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 1 -- start start 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 1 librados: setting wanted keys 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 1 librados: calling monclient init 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.507+0000 7fa10b14b640 10 monclient: init 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:47.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient(hunting): picked mon.noname-a con 0x7fa1040b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient(hunting): picked mon.noname-b con 0x7fa1040b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient(hunting): picked mon.noname-c con 0x7fa10414be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa104140720 con 0x7fa1040b1730 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa104140380 con 0x7fa1040b4d60 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa104145730 con 0x7fa10414be90 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa10b14b640 10 monclient(hunting): authenticate will time out at 7267.770925s 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa1096e8640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fa10414be90 0x7fa1041e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:35070/0 (socket says 172.21.15.132:35070) 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa1096e8640 1 -- 172.21.15.132:0/284520636 learned_addr learned my addr 172.21.15.132:0/284520636 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa1017fa640 10 client.?.objecter ms_handle_connect 0x7fa10414be90 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa1017fa640 10 client.?.objecter ms_handle_connect 0x7fa1040b1730 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.508+0000 7fa1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:47.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.509+0000 7fa1017fa640 10 client.?.objecter ms_handle_connect 0x7fa1040b4d60 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.509+0000 7fa1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.510+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169430024 0 0) 0x7fa104140380 con 0x7fa1040b4d60 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.510+0000 7fa1017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.510+0000 7fa1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.510+0000 7fa1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.510+0000 7fa1017fa640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.510+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e8004eb0 con 0x7fa1040b4d60 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224236210 0 0) 0x7fa104145730 con 0x7fa10414be90 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): my global_id is 4199 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104140380 con 0x7fa10414be90 2024-03-25T18:13:47.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870912996 0 0) 0x7fa104140720 con 0x7fa1040b1730 2024-03-25T18:13:47.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:47.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:47.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:47.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:13:47.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.511+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104145730 con 0x7fa1040b1730 2024-03-25T18:13:47.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4182344419 0 0) 0x7fa0e8004eb0 con 0x7fa1040b4d60 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa104140720 con 0x7fa1040b4d60 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa0f0001930 con 0x7fa1040b4d60 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 10 monclient(hunting): dump: 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:47.516 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.515+0000 7fa1017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.516+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2679967525 0 0) 0x7fa104140380 con 0x7fa10414be90 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.516+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e8004eb0 con 0x7fa10414be90 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.516+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa0fc001b10 con 0x7fa10414be90 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.516+0000 7fa1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.516+0000 7fa1017fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:47.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.516+0000 7fa1017fa640 10 monclient(hunting): dump: 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:47.518 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.517+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678187707 0 0) 0x7fa104145730 con 0x7fa1040b1730 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.517+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa104140380 con 0x7fa1040b1730 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.517+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa0f8001b00 con 0x7fa1040b1730 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.517+0000 7fa1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.517+0000 7fa1017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.517+0000 7fa1017fa640 10 monclient(hunting): dump: 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.520 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2225970428 0 0) 0x7fa104140720 con 0x7fa1040b4d60 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 >> v1:172.21.15.132:6790/0 conn(0x7fa10414be90 legacy=0x7fa1041e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 >> v1:172.21.15.132:6789/0 conn(0x7fa1040b1730 legacy=0x7fa104131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 1 monclient: found mon.b 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1041458c0 con 0x7fa1040b4d60 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:47.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa1017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 10 monclient: _renew_subs 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa104144c00 con 0x7fa1040b4d60 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 10 monclient: _renew_subs 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10413e030 con 0x7fa1040b4d60 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.518+0000 7fa10b14b640 1 librados: init done 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.519+0000 7fa10b14b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.519+0000 7fa10b14b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.519+0000 7fa10b14b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:47.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.519+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104144f00 con 0x7fa1040b4d60 2024-03-25T18:13:47.529 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:10.012 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f00047e0 con 0x7fa1040b4d60 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa1017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa0f0004950 con 0x7fa1040b4d60 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa1017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa1017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa1017fa640 10 monclient: dump: 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.534 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.533+0000 7fa102ffd640 4 set_mon_vals no callback set 2024-03-25T18:13:47.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.534+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fa0f001a920 con 0x7fa1040b4d60 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.534+0000 7fa108ee7640 10 monclient: get_auth_request con 0x7fa0e806e060 auth_method 0 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.534+0000 7fa1017fa640 10 client.4207.objecter ms_handle_connect 0x7fa0e806e060 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.535+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fa0f0054600 con 0x7fa1040b4d60 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.535+0000 7fa1017fa640 10 client.4207.objecter ms_dispatch 0x7fa104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.535+0000 7fa1017fa640 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.535+0000 7fa1017fa640 3 client.4207.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:47.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.535+0000 7fa1017fa640 20 client.4207.objecter dump_active .. 0 homeless 2024-03-25T18:13:47.572 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:10.052 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.578 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:10.064 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.623 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:10.102 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.729 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:10.195 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.745 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:10.226 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.750 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:10.236 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.770+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0f004ff40 con 0x7fa1040b4d60 2024-03-25T18:13:47.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.770+0000 7fa1017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:47.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.770+0000 7fa1017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:47.777 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:10.245 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.807 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:10.289 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.842 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:10.328 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:47.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.943+0000 7fa10b14b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.943+0000 7fa10b14b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:47.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.943+0000 7fa10b14b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:13:47.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:47.943+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1041408b0 con 0x7fa1040b4d60 2024-03-25T18:13:48.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.000+0000 7fa1017fa640 1 -- 172.21.15.132:0/284520636 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fa0f0052040 con 0x7fa1040b4d60 2024-03-25T18:13:48.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.000+0000 7fa1017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:48.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.000+0000 7fa1017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:48.001 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:13:48.001 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:48.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 10 librados: watch_flush enter 2024-03-25T18:13:48.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 10 librados: watch_flush exit 2024-03-25T18:13:48.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 20 client.4207.objecter shutdown clearing up homeless session... 2024-03-25T18:13:48.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 10 client.4207.objecter successfully canceled tick 2024-03-25T18:13:48.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 >> v1:172.21.15.161:6800/411950978 conn(0x7fa0e806e060 legacy=0x7fa0e8070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:48.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 10 monclient: shutdown 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 >> v1:172.21.15.161:6789/0 conn(0x7fa1040b4d60 legacy=0x7fa1041daca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 shutdown_connections 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 >> 172.21.15.132:0/284520636 conn(0x7fa1040617a0 msgr2=0x7fa104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 shutdown_connections 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 -- 172.21.15.132:0/284520636 wait complete. 2024-03-25T18:13:48.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:48.002+0000 7fa10b14b640 1 librados: shutdown 2024-03-25T18:13:48.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:48.029 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:10.495 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.045 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:10.532 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.055 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:10.534 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.067 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:10.535 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.075 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:10.557 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.108 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:10.594 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.124 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:10.603 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.142 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:10.621 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.197 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:10.677 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.247 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:10.728 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.317 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:10.783 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.317 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:10.798 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.376 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:10.844 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.575 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:11.058 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.597 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:11.082 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.602 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:11.069 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.621 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:11.107 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.629 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:11.112 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.651 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:11.137 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.687 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:11.174 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.899 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:11.366 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:48.954 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:11.421 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:49.191 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:11.658 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:49.248 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:11.715 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:53.443 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:15.922 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:53.447 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:15.926 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:53.718 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:16.200 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:53.808 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:16.294 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:54.052 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:16.538 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:54.271 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:16.753 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:54.406 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:16.873 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:54.818 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:54.881 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:17.347 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:54.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.923+0000 7f21c4904640 10 monclient: get_monmap_and_config 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.923+0000 7f21c4904640 10 monclient: build_initial_monmap 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.923+0000 7f21c4904640 10 monclient: monmap: 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:13:54.924698+0000 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:13:54.924698+0000 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.924+0000 7f21c4904640 1 Processor -- start 2024-03-25T18:13:54.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.924+0000 7f21c4904640 1 -- start start 2024-03-25T18:13:54.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.924+0000 7f21c4904640 10 monclient: init 2024-03-25T18:13:54.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:54.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:54.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient(hunting): picked mon.noname-b con 0x7f21c00b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:54.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient(hunting): picked mon.noname-c con 0x7f21c00b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:54.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient(hunting): picked mon.noname-a con 0x7f21c014be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:54.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c00af940 con 0x7f21c014be90 2024-03-25T18:13:54.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c00b0a70 con 0x7f21c00b1730 2024-03-25T18:13:54.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c00b0d50 con 0x7f21c00b4d60 2024-03-25T18:13:54.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:54.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21c4904640 10 monclient(hunting): authenticate will time out at 7275.188340s 2024-03-25T18:13:54.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21bed9d640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f21c014be90 0x7f21c014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:52696/0 (socket says 172.21.15.132:52696) 2024-03-25T18:13:54.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.925+0000 7f21bed9d640 1 -- 172.21.15.132:0/2642246285 learned_addr learned my addr 172.21.15.132:0/2642246285 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:54.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.928+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730630170 0 0) 0x7f21c00af940 con 0x7f21c014be90 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.928+0000 7f21bd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.928+0000 7f21bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.928+0000 7f21bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.928+0000 7f21bd59a640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.928+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a8003520 con 0x7f21c014be90 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315321086 0 0) 0x7f21c00b0d50 con 0x7f21c00b4d60 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c00af940 con 0x7f21c00b4d60 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075212309 0 0) 0x7f21c00b0a70 con 0x7f21c00b1730 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.929+0000 7f21bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.930+0000 7f21bd59a640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:13:54.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.930+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c00b0d50 con 0x7f21c00b1730 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1417613891 0 0) 0x7f21c00b0d50 con 0x7f21c00b1730 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 >> v1:172.21.15.132:6790/0 conn(0x7f21c00b4d60 legacy=0x7f21c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 >> v1:172.21.15.132:6789/0 conn(0x7f21c014be90 legacy=0x7f21c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 1 monclient: found mon.noname-c 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00b1030 con 0x7f21c00b1730 2024-03-25T18:13:54.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: _finish_auth 0 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21c4904640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21c4904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f21b40014f0 con 0x7f21c00b1730 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21bd59a640 10 monclient: dump: 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:54.937 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:54.938 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:54.938 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.934+0000 7f21c4904640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:54.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.936+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21b40017c0 con 0x7f21c00b1730 2024-03-25T18:13:54.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.936+0000 7f21bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:54.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.936+0000 7f21c4904640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:54.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.936+0000 7f21c4904640 4 set_mon_vals no callback set 2024-03-25T18:13:54.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21bd59a640 1 -- 172.21.15.132:0/2642246285 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f21b4001b50 con 0x7f21c00b1730 2024-03-25T18:13:54.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21bd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21bd59a640 10 monclient: dump: 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21c4904640 10 monclient: shutdown 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21c4904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21c4904640 1 -- 172.21.15.132:0/2642246285 >> v1:172.21.15.161:6789/0 conn(0x7f21c00b1730 legacy=0x7f21c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21c4904640 1 -- 172.21.15.132:0/2642246285 shutdown_connections 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.937+0000 7f21c4904640 1 -- 172.21.15.132:0/2642246285 >> 172.21.15.132:0/2642246285 conn(0x7f21c00617a0 msgr2=0x7f21c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.938+0000 7f21c4904640 1 -- 172.21.15.132:0/2642246285 shutdown_connections 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.938+0000 7f21c4904640 1 -- 172.21.15.132:0/2642246285 wait complete. 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.939+0000 7f21c4904640 10 monclient: build_initial_monmap 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.939+0000 7f21c4904640 10 monclient: monmap: 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:13:54.941 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.939+0000 7f21c4904640 1 librados: starting msgr at 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.939+0000 7f21c4904640 1 librados: starting objecter 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.939+0000 7f21c4904640 1 Processor -- start 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.940+0000 7f21c4904640 1 -- start start 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.940+0000 7f21c4904640 1 librados: setting wanted keys 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.940+0000 7f21c4904640 1 librados: calling monclient init 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.940+0000 7f21c4904640 10 monclient: init 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient(hunting): picked mon.noname-a con 0x7f21c00b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient(hunting): picked mon.noname-b con 0x7f21c00b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient(hunting): picked mon.noname-c con 0x7f21c014be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c0140ec0 con 0x7f21c00b1730 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c01388e0 con 0x7f21c00b4d60 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c0137ef0 con 0x7f21c014be90 2024-03-25T18:13:54.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21c4904640 10 monclient(hunting): authenticate will time out at 7275.204188s 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.941+0000 7f21bdd9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f21c00b4d60 0x7f21c01d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:47208/0 (socket says 172.21.15.132:47208) 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.942+0000 7f21bdd9b640 1 -- 172.21.15.132:0/4117971878 learned_addr learned my addr 172.21.15.132:0/4117971878 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.942+0000 7f21aeffd640 10 client.?.objecter ms_handle_connect 0x7f21c00b4d60 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.942+0000 7f21aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.943+0000 7f21aeffd640 10 client.?.objecter ms_handle_connect 0x7f21c014be90 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.943+0000 7f21aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.943+0000 7f21aeffd640 10 client.?.objecter ms_handle_connect 0x7f21c00b1730 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.943+0000 7f21aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:54.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.944+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839508350 0 0) 0x7f21c0137ef0 con 0x7f21c014be90 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a0004eb0 con 0x7f21c014be90 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673749747 0 0) 0x7f21c01388e0 con 0x7f21c00b4d60 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:13:54.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c0137ef0 con 0x7f21c00b4d60 2024-03-25T18:13:54.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561525681 0 0) 0x7f21c0140ec0 con 0x7f21c00b1730 2024-03-25T18:13:54.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:54.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:54.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:54.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:13:54.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.945+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c01388e0 con 0x7f21c00b1730 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2346407133 0 0) 0x7f21c01388e0 con 0x7f21c00b1730 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21a0005620 con 0x7f21c00b1730 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f21b0001930 con 0x7f21c00b1730 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 10 monclient(hunting): dump: 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.954 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.951+0000 7f21aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3495992233 0 0) 0x7f21c0137ef0 con 0x7f21c00b4d60 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21a0005c80 con 0x7f21c00b4d60 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f21b40015b0 con 0x7f21c00b4d60 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 10 monclient(hunting): dump: 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.955 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2794607817 0 0) 0x7f21a0004eb0 con 0x7f21c014be90 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21c0137ef0 con 0x7f21c014be90 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f21b8001b30 con 0x7f21c014be90 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.952+0000 7f21aeffd640 10 monclient(hunting): dump: 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.956 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2548207361 0 0) 0x7f21a0005620 con 0x7f21c00b1730 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 >> v1:172.21.15.132:6790/0 conn(0x7f21c014be90 legacy=0x7f21c01e2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 >> v1:172.21.15.161:6789/0 conn(0x7f21c00b4d60 legacy=0x7f21c01d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 1 monclient: found mon.a 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0138080 con 0x7f21c00b1730 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21c4904640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21c4904640 10 monclient: _renew_subs 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21c4904640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.955+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c0140380 con 0x7f21c00b1730 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.956+0000 7f21c4904640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.956+0000 7f21c4904640 10 monclient: _renew_subs 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.956+0000 7f21c4904640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:54.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.956+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c0145330 con 0x7f21c00b1730 2024-03-25T18:13:54.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.956+0000 7f21c4904640 1 librados: init done 2024-03-25T18:13:54.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.957+0000 7f21c4904640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.957+0000 7f21c4904640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:54.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.957+0000 7f21c4904640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:54.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.957+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c00b2270 con 0x7f21c00b1730 2024-03-25T18:13:54.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.958+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21b00035f0 con 0x7f21c00b1730 2024-03-25T18:13:54.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.958+0000 7f21aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:54.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.958+0000 7f21bcd99640 4 set_mon_vals no callback set 2024-03-25T18:13:54.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.959+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f21b0001930 con 0x7f21c00b1730 2024-03-25T18:13:54.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.959+0000 7f21aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:54.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.959+0000 7f21aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:54.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:54.959+0000 7f21aeffd640 10 monclient: dump: 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:13:54.961 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:13:55.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.370+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f21b00195d0 con 0x7f21c00b1730 2024-03-25T18:13:55.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.371+0000 7f21bdd9b640 10 monclient: get_auth_request con 0x7f21a006e800 auth_method 0 2024-03-25T18:13:55.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.371+0000 7f21aeffd640 10 client.4212.objecter ms_handle_connect 0x7f21a006e800 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f21b0052f60 con 0x7f21c00b1730 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 10 client.4212.objecter ms_dispatch 0x7f21c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21b004f200 con 0x7f21c00b1730 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:55.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.379+0000 7f21aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:55.676 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.675+0000 7f21c4904640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:55.676 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.675+0000 7f21c4904640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:55.676 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.675+0000 7f21c4904640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:13:55.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.675+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21c00627b0 con 0x7f21c00b1730 2024-03-25T18:13:55.700 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.699+0000 7f21aeffd640 1 -- 172.21.15.132:0/4117971878 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f21b004f3a0 con 0x7f21c00b1730 2024-03-25T18:13:55.700 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.699+0000 7f21aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:55.700 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.699+0000 7f21aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:55.700 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:13:55.700 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:55.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 10 librados: watch_flush enter 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 10 librados: watch_flush exit 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 >> v1:172.21.15.161:6800/411950978 conn(0x7f21a006e800 legacy=0x7f21a0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 10 monclient: shutdown 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 >> v1:172.21.15.132:6789/0 conn(0x7f21c00b1730 legacy=0x7f21c0131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 shutdown_connections 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 >> 172.21.15.132:0/4117971878 conn(0x7f21c00617a0 msgr2=0x7f21c00b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 shutdown_connections 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.701+0000 7f21c4904640 1 -- 172.21.15.132:0/4117971878 wait complete. 2024-03-25T18:13:55.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:13:55.702+0000 7f21c4904640 1 librados: shutdown 2024-03-25T18:13:55.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:58.998 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:21.476 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.001 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:21.480 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.049 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:21.528 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.053 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:21.532 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.205 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:21.691 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.211 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:21.697 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.267 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:21.749 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.271 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:21.753 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.988 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:22.475 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:13:59.994 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:22.481 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:00.217 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:22.699 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:00.220 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:22.703 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:00.289 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:22.756 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:00.293 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:22.760 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:00.310 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:22.776 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:00.314 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:22.781 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:02.520 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.609+0000 7f2bcc115640 10 monclient: get_monmap_and_config 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.609+0000 7f2bcc115640 10 monclient: build_initial_monmap 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.609+0000 7f2bcc115640 10 monclient: monmap: 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:02.610477+0000 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:02.610477+0000 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:02.610 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:02.611 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:02.611 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:02.611 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:02.611 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.609+0000 7f2bcc115640 1 Processor -- start 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 1 -- start start 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient: init 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc40b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc40b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc414be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:02.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40af940 con 0x7f2bc40b4d60 2024-03-25T18:14:02.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40b0a70 con 0x7f2bc414be90 2024-03-25T18:14:02.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc40b0d50 con 0x7f2bc40b1730 2024-03-25T18:14:02.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:02.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.610+0000 7f2bcc115640 10 monclient(hunting): authenticate will time out at 7282.873669s 2024-03-25T18:14:02.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.611+0000 7f2bc96b0640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f2bc40b1730 0x7f2bc40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:46288/0 (socket says 172.21.15.132:46288) 2024-03-25T18:14:02.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.611+0000 7f2bc96b0640 1 -- 172.21.15.132:0/4012636199 learned_addr learned my addr 172.21.15.132:0/4012636199 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:02.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062244288 0 0) 0x7f2bc40b0d50 con 0x7f2bc40b1730 2024-03-25T18:14:02.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:14:02.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bb0003520 con 0x7f2bc40b1730 2024-03-25T18:14:02.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222993940 0 0) 0x7f2bc40af940 con 0x7f2bc40b4d60 2024-03-25T18:14:02.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc40b0d50 con 0x7f2bc40b4d60 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340040493 0 0) 0x7f2bc40b0a70 con 0x7f2bc414be90 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:14:02.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.614+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc40af940 con 0x7f2bc414be90 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737496541 0 0) 0x7f2bb0003520 con 0x7f2bc40b1730 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 >> v1:172.21.15.161:6789/0 conn(0x7f2bc414be90 legacy=0x7f2bc414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 >> v1:172.21.15.132:6789/0 conn(0x7f2bc40b4d60 legacy=0x7f2bc413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: _finish_hunting 0 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 1 monclient: found mon.noname-b 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc40b1030 con 0x7f2bc40b1730 2024-03-25T18:14:02.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: _finish_auth 0 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: _check_auth_tickets 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2bb4001690 con 0x7f2bc40b1730 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bc8eaf640 10 monclient: dump: 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:02.620 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:02.621 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:02.621 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bcc115640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:14:02.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.618+0000 7f2bcc115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:02.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bb40017f0 con 0x7f2bc40b1730 2024-03-25T18:14:02.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bc8eaf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:02.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bc8eaf640 1 -- 172.21.15.132:0/4012636199 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2bb4001c00 con 0x7f2bc40b1730 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bc8eaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bc8eaf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bc8eaf640 10 monclient: dump: 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:02.623 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bcc115640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.620+0000 7f2bcc115640 4 set_mon_vals no callback set 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.621+0000 7f2bcc115640 10 monclient: shutdown 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.621+0000 7f2bcc115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.621+0000 7f2bcc115640 1 -- 172.21.15.132:0/4012636199 >> v1:172.21.15.132:6790/0 conn(0x7f2bc40b1730 legacy=0x7f2bc40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.621+0000 7f2bcc115640 1 -- 172.21.15.132:0/4012636199 shutdown_connections 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.621+0000 7f2bcc115640 1 -- 172.21.15.132:0/4012636199 >> 172.21.15.132:0/4012636199 conn(0x7f2bc40617a0 msgr2=0x7f2bc4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.621+0000 7f2bcc115640 1 -- 172.21.15.132:0/4012636199 shutdown_connections 2024-03-25T18:14:02.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.622+0000 7f2bcc115640 1 -- 172.21.15.132:0/4012636199 wait complete. 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.622+0000 7f2bcc115640 10 monclient: build_initial_monmap 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.622+0000 7f2bcc115640 10 monclient: monmap: 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 1 librados: starting msgr at 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 1 librados: starting objecter 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 1 Processor -- start 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 1 -- start start 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 1 librados: setting wanted keys 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 1 librados: calling monclient init 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.623+0000 7f2bcc115640 10 monclient: init 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient(hunting): picked mon.noname-b con 0x7f2bc40b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient(hunting): picked mon.noname-c con 0x7f2bc40b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient(hunting): picked mon.noname-a con 0x7f2bc414be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4141bf0 con 0x7f2bc414be90 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4141ed0 con 0x7f2bc40b1730 2024-03-25T18:14:02.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc41421b0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.624+0000 7f2bcc115640 10 monclient(hunting): authenticate will time out at 7282.887394s 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bc96b0640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f2bc40b4d60 0x7f2bc4133ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:46298/0 (socket says 172.21.15.132:46298) 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bc96b0640 1 -- 172.21.15.132:0/3570768455 learned_addr learned my addr 172.21.15.132:0/3570768455 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bba7fc640 10 client.?.objecter ms_handle_connect 0x7f2bc40b4d60 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bba7fc640 10 client.?.objecter ms_handle_connect 0x7f2bc40b1730 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bba7fc640 10 client.?.objecter ms_handle_connect 0x7f2bc414be90 2024-03-25T18:14:02.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.625+0000 7f2bba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:02.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.627+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702468929 0 0) 0x7f2bc41421b0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.627+0000 7f2bba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.627+0000 7f2bba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.627+0000 7f2bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.627+0000 7f2bba7fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.627+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ba8004eb0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054308051 0 0) 0x7f2bc4141ed0 con 0x7f2bc40b1730 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc41421b0 con 0x7f2bc40b1730 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345582414 0 0) 0x7f2bc4141bf0 con 0x7f2bc414be90 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:14:02.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.628+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc4141ed0 con 0x7f2bc414be90 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2765449500 0 0) 0x7f2ba8004eb0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc4141bf0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2bb4001bc0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.633 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:02.634 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:02.634 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:02.634 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:02.634 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.632+0000 7f2bba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.635+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3391300380 0 0) 0x7f2bc41421b0 con 0x7f2bc40b1730 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.635+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ba8004eb0 con 0x7f2bc40b1730 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.635+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2bc0001990 con 0x7f2bc40b1730 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.635+0000 7f2bba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.635+0000 7f2bba7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.635+0000 7f2bba7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.636 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:02.637 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:02.637 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:02.637 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:02.637 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 433372881 0 0) 0x7f2bc4141ed0 con 0x7f2bc414be90 2024-03-25T18:14:02.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc41421b0 con 0x7f2bc414be90 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2290961160 0 0) 0x7f2bc4141bf0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 >> v1:172.21.15.161:6789/0 conn(0x7f2bc40b1730 legacy=0x7f2bc41425d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 >> v1:172.21.15.132:6789/0 conn(0x7f2bc414be90 legacy=0x7f2bc41341f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 monclient: found mon.c 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc4140f00 con 0x7f2bc40b4d60 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:02.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 5 monclient: authenticate success, global_id 4211 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 10 monclient: _renew_subs 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc41406f0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 10 monclient: _renew_subs 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc4140380 con 0x7f2bc40b4d60 2024-03-25T18:14:02.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.636+0000 7f2bcc115640 1 librados: init done 2024-03-25T18:14:02.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.637+0000 7f2bcc115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.637+0000 7f2bcc115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.637+0000 7f2bcc115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:14:02.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.637+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc41401f0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.637+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bb40011f0 con 0x7f2bc40b4d60 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.638+0000 7f2bba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.638+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2bb4004c30 con 0x7f2bc40b4d60 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.638+0000 7f2bba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.638+0000 7f2bba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.638+0000 7f2bba7fc640 10 monclient: dump: 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:02.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.638+0000 7f2bbbfff640 4 set_mon_vals no callback set 2024-03-25T18:14:02.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.653+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f2bb401ac00 con 0x7f2bc40b4d60 2024-03-25T18:14:02.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.654+0000 7f2bc9eb1640 10 monclient: get_auth_request con 0x7f2ba806df30 auth_method 0 2024-03-25T18:14:02.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.655+0000 7f2bba7fc640 10 client.4211.objecter ms_handle_connect 0x7f2ba806df30 2024-03-25T18:14:02.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.656+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f2bb4054930 con 0x7f2bc40b4d60 2024-03-25T18:14:02.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.656+0000 7f2bba7fc640 10 client.4211.objecter ms_dispatch 0x7f2bc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:02.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.656+0000 7f2bba7fc640 3 client.4211.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:02.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.656+0000 7f2bba7fc640 3 client.4211.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:02.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.656+0000 7f2bba7fc640 20 client.4211.objecter dump_active .. 0 homeless 2024-03-25T18:14:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.895+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bb4050270 con 0x7f2bc40b4d60 2024-03-25T18:14:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.895+0000 7f2bba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:02.895+0000 7f2bba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:03.043 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:00:25.521 35309== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:03.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.181+0000 7f2bcc115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:03.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.181+0000 7f2bcc115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:03.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.181+0000 7f2bcc115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:14:03.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.181+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bc40627b0 con 0x7f2bc40b4d60 2024-03-25T18:14:03.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.207+0000 7f2bba7fc640 1 -- 172.21.15.132:0/3570768455 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f2bb4052370 con 0x7f2bc40b4d60 2024-03-25T18:14:03.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.207+0000 7f2bba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:03.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.207+0000 7f2bba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:03.208 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:03.208 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 10 librados: watch_flush enter 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 10 librados: watch_flush exit 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 20 client.4211.objecter shutdown clearing up homeless session... 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 10 client.4211.objecter successfully canceled tick 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 >> v1:172.21.15.161:6800/411950978 conn(0x7f2ba806df30 legacy=0x7f2ba80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 10 monclient: shutdown 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 >> v1:172.21.15.132:6790/0 conn(0x7f2bc40b4d60 legacy=0x7f2bc4133ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:03.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 shutdown_connections 2024-03-25T18:14:03.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 >> 172.21.15.132:0/3570768455 conn(0x7f2bc40617a0 msgr2=0x7f2bc413c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:03.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 shutdown_connections 2024-03-25T18:14:03.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 -- 172.21.15.132:0/3570768455 wait complete. 2024-03-25T18:14:03.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:03.210+0000 7f2bcc115640 1 librados: shutdown 2024-03-25T18:14:03.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:03.320 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:00:25.805 35307== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:03.331 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:00:25.810 35310== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:03.556 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:00:26.037 35308== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:04.031 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:00:26.517 35202== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:04.391 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:00:26.858 35205== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:04.571 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:00:27.053 35203== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:04.575 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:00:27.041 35204== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:10.031 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:10.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.125+0000 7f94a03f6640 10 monclient: get_monmap_and_config 2024-03-25T18:14:10.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.125+0000 7f94a03f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:10.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.125+0000 7f94a03f6640 10 monclient: monmap: 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:10.126768+0000 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:10.126768+0000 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:10.127 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.126+0000 7f94a03f6640 1 Processor -- start 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.126+0000 7f94a03f6640 1 -- start start 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.126+0000 7f94a03f6640 10 monclient: init 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient(hunting): picked mon.noname-a con 0x7f94980a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient(hunting): picked mon.noname-b con 0x7f949813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient(hunting): picked mon.noname-c con 0x7f94981430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9498142520 con 0x7f94980a98f0 2024-03-25T18:14:10.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94980b28a0 con 0x7f94981430c0 2024-03-25T18:14:10.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94980b2a30 con 0x7f949813f710 2024-03-25T18:14:10.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:10.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.127+0000 7f94a03f6640 10 monclient(hunting): authenticate will time out at 7290.390657s 2024-03-25T18:14:10.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.128+0000 7f949e993640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f94981430c0 0x7f949814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:42416/0 (socket says 172.21.15.132:42416) 2024-03-25T18:14:10.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.128+0000 7f949e993640 1 -- 172.21.15.132:0/1163542505 learned_addr learned my addr 172.21.15.132:0/1163542505 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.200+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233680951 0 0) 0x7f94980b28a0 con 0x7f94981430c0 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.200+0000 7f949d190640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.200+0000 7f949d190640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.200+0000 7f949d190640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.200+0000 7f949d190640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.200+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9484003520 con 0x7f94981430c0 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799628199 0 0) 0x7f94980b2a30 con 0x7f949813f710 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:14:10.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94980b28a0 con 0x7f949813f710 2024-03-25T18:14:10.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082594908 0 0) 0x7f9498142520 con 0x7f94980a98f0 2024-03-25T18:14:10.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:14:10.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.201+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94980b2a30 con 0x7f94980a98f0 2024-03-25T18:14:10.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 715424995 0 0) 0x7f9484003520 con 0x7f94981430c0 2024-03-25T18:14:10.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 >> v1:172.21.15.132:6790/0 conn(0x7f949813f710 legacy=0x7f94981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 >> v1:172.21.15.132:6789/0 conn(0x7f94980a98f0 legacy=0x7f94980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: _finish_hunting 0 2024-03-25T18:14:10.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 1 monclient: found mon.noname-c 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980b44c0 con 0x7f94981430c0 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: _finish_auth 0 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: _check_auth_tickets 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9490001a10 con 0x7f94981430c0 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f949d190640 10 monclient: dump: 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.207 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f94a03f6640 5 monclient: authenticate success, global_id 4222 2024-03-25T18:14:10.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.205+0000 7f94a03f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9490001b70 con 0x7f94981430c0 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f949d190640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f949d190640 1 -- 172.21.15.132:0/1163542505 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9490001f80 con 0x7f94981430c0 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f94a03f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f94a03f6640 4 set_mon_vals no callback set 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f949d190640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f949d190640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.208+0000 7f949d190640 10 monclient: dump: 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.209 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.210 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:10.210 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:10.210 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:10.210 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 10 monclient: shutdown 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 1 -- 172.21.15.132:0/1163542505 >> v1:172.21.15.161:6789/0 conn(0x7f94981430c0 legacy=0x7f949814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 1 -- 172.21.15.132:0/1163542505 shutdown_connections 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 1 -- 172.21.15.132:0/1163542505 >> 172.21.15.132:0/1163542505 conn(0x7f94980b7ea0 msgr2=0x7f94980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 1 -- 172.21.15.132:0/1163542505 shutdown_connections 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.209+0000 7f94a03f6640 1 -- 172.21.15.132:0/1163542505 wait complete. 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.210+0000 7f94a03f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.210+0000 7f94a03f6640 10 monclient: monmap: 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:10.211 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:10.212 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 1 librados: starting msgr at 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 1 librados: starting objecter 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 1 Processor -- start 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 1 -- start start 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 1 librados: setting wanted keys 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 1 librados: calling monclient init 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.211+0000 7f94a03f6640 10 monclient: init 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:10.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient(hunting): picked mon.noname-c con 0x7f94980a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient(hunting): picked mon.noname-a con 0x7f949813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient(hunting): picked mon.noname-b con 0x7f94981430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9498136f60 con 0x7f949813f710 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94981370f0 con 0x7f94981430c0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949813b640 con 0x7f94980a98f0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f94a03f6640 10 monclient(hunting): authenticate will time out at 7290.475367s 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f949e192640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f94980a98f0 0x7f9498139470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:46316/0 (socket says 172.21.15.132:46316) 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.212+0000 7f949e192640 1 -- 172.21.15.132:0/3339920925 learned_addr learned my addr 172.21.15.132:0/3339920925 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.213+0000 7f948affd640 10 client.?.objecter ms_handle_connect 0x7f94980a98f0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.213+0000 7f948affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.213+0000 7f948affd640 10 client.?.objecter ms_handle_connect 0x7f94981430c0 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.213+0000 7f948affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:10.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.213+0000 7f948affd640 10 client.?.objecter ms_handle_connect 0x7f949813f710 2024-03-25T18:14:10.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.213+0000 7f948affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.215+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709456730 0 0) 0x7f94981370f0 con 0x7f94981430c0 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.215+0000 7f948affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.215+0000 7f948affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.215+0000 7f948affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.215+0000 7f948affd640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.215+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f947c004eb0 con 0x7f94981430c0 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.216+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331358269 0 0) 0x7f949813b640 con 0x7f94980a98f0 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.216+0000 7f948affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.216+0000 7f948affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.216+0000 7f948affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.216+0000 7f948affd640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:14:10.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.216+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f947c0051d0 con 0x7f94980a98f0 2024-03-25T18:14:10.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.217+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051806534 0 0) 0x7f9498136f60 con 0x7f949813f710 2024-03-25T18:14:10.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.217+0000 7f948affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:10.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.217+0000 7f948affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:10.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.217+0000 7f948affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:10.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.217+0000 7f948affd640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:14:10.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.217+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f949813b640 con 0x7f949813f710 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352920891 0 0) 0x7f947c004eb0 con 0x7f94981430c0 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f947c0057d0 con 0x7f94981430c0 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9490001e80 con 0x7f94981430c0 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): dump: 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:10.224 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:10.225 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:10.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 512094832 0 0) 0x7f947c0051d0 con 0x7f94980a98f0 2024-03-25T18:14:10.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f947c005ea0 con 0x7f94980a98f0 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f948c0019f0 con 0x7f94980a98f0 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.223+0000 7f948affd640 10 monclient(hunting): dump: 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:10.226 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.227+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3589203936 0 0) 0x7f949813b640 con 0x7f949813f710 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.227+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f947c0051d0 con 0x7f949813f710 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.227+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f94940018f0 con 0x7f949813f710 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.227+0000 7f948affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.227+0000 7f948affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.227+0000 7f948affd640 10 monclient(hunting): dump: 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:10.228 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:10.229 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 612818520 0 0) 0x7f947c0057d0 con 0x7f94981430c0 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 >> v1:172.21.15.132:6790/0 conn(0x7f94980a98f0 legacy=0x7f9498139470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 >> v1:172.21.15.132:6789/0 conn(0x7f949813f710 legacy=0x7f9498135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 1 monclient: found mon.b 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949813b7d0 con 0x7f94981430c0 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:10.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f948affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f94a03f6640 5 monclient: authenticate success, global_id 4225 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f94a03f6640 10 monclient: _renew_subs 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f94a03f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.228+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94981379a0 con 0x7f94981430c0 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.229+0000 7f94a03f6640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.229+0000 7f94a03f6640 10 monclient: _renew_subs 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.229+0000 7f94a03f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.229+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94981e3760 con 0x7f94981430c0 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.229+0000 7f94a03f6640 1 librados: init done 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.230+0000 7f94a03f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.230+0000 7f94a03f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.230+0000 7f94a03f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:10.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.230+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9498137e80 con 0x7f94981430c0 2024-03-25T18:14:10.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.449+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9490001560 con 0x7f94981430c0 2024-03-25T18:14:10.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.449+0000 7f948affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:10.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.449+0000 7f949c98f640 4 set_mon_vals no callback set 2024-03-25T18:14:10.665 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.664+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9490001e80 con 0x7f94981430c0 2024-03-25T18:14:10.665 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.664+0000 7f948affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:10.665 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.664+0000 7f948affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.664+0000 7f948affd640 10 monclient: dump: 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:10.666 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:10.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.665+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f949001ac80 con 0x7f94981430c0 2024-03-25T18:14:10.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.666+0000 7f949e192640 10 monclient: get_auth_request con 0x7f947c06eac0 auth_method 0 2024-03-25T18:14:10.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.666+0000 7f948affd640 10 client.4225.objecter ms_handle_connect 0x7f947c06eac0 2024-03-25T18:14:10.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.670+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f9490054770 con 0x7f94981430c0 2024-03-25T18:14:10.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.670+0000 7f948affd640 10 client.4225.objecter ms_dispatch 0x7f9498000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:10.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.670+0000 7f948affd640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:10.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.670+0000 7f948affd640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:10.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.670+0000 7f948affd640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-25T18:14:10.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.673+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9490050280 con 0x7f94981430c0 2024-03-25T18:14:10.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.673+0000 7f948affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:10.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.673+0000 7f948affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:10.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.862+0000 7f94a03f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.862+0000 7f94a03f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.862+0000 7f94a03f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:10.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.862+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94980627b0 con 0x7f94981430c0 2024-03-25T18:14:10.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.886+0000 7f948affd640 1 -- 172.21.15.132:0/3339920925 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f94900521b0 con 0x7f94981430c0 2024-03-25T18:14:10.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.887+0000 7f948affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:10.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.887+0000 7f948affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:10.888 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:10.888 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.889+0000 7f94a03f6640 10 librados: watch_flush enter 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.889+0000 7f94a03f6640 10 librados: watch_flush exit 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.889+0000 7f94a03f6640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.889+0000 7f94a03f6640 10 client.4225.objecter successfully canceled tick 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.889+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 >> v1:172.21.15.161:6800/411950978 conn(0x7f947c06eac0 legacy=0x7f947c070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 10 monclient: shutdown 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 >> v1:172.21.15.161:6789/0 conn(0x7f94981430c0 legacy=0x7f94981e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 shutdown_connections 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 >> 172.21.15.132:0/3339920925 conn(0x7f94980b7ea0 msgr2=0x7f9498074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:10.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 shutdown_connections 2024-03-25T18:14:10.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 1 -- 172.21.15.132:0/3339920925 wait complete. 2024-03-25T18:14:10.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:10.890+0000 7f94a03f6640 1 librados: shutdown 2024-03-25T18:14:10.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:11.657 INFO:tasks.ceph.osd.7.smithi161.stderr:2024-03-25T18:14:11.647+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:14:11.677 INFO:tasks.ceph.osd.6.smithi161.stderr:2024-03-25T18:14:11.672+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:14:12.005 INFO:tasks.ceph.osd.4.smithi161.stderr:2024-03-25T18:14:12.001+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:14:12.045 INFO:tasks.ceph.osd.5.smithi161.stderr:2024-03-25T18:14:12.040+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:14:12.584 INFO:tasks.ceph.osd.0.smithi132.stderr:2024-03-25T18:14:12.580+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:14:13.110 INFO:tasks.ceph.osd.1.smithi132.stderr:2024-03-25T18:14:13.105+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:14:13.348 INFO:tasks.ceph.osd.3.smithi132.stderr:2024-03-25T18:14:13.345+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:14:13.429 INFO:tasks.ceph.osd.2.smithi132.stderr:2024-03-25T18:14:13.424+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:14:17.711 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:17.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.827+0000 7f29e937b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:17.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.827+0000 7f29e937b640 10 monclient: build_initial_monmap 2024-03-25T18:14:17.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.827+0000 7f29e937b640 10 monclient: monmap: 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:17.828976+0000 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:17.828976+0000 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:17.829 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:17.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.828+0000 7f29e937b640 1 Processor -- start 2024-03-25T18:14:17.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.828+0000 7f29e937b640 1 -- start start 2024-03-25T18:14:17.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.828+0000 7f29e937b640 10 monclient: init 2024-03-25T18:14:17.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient(hunting): picked mon.noname-b con 0x7f29e40a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient(hunting): picked mon.noname-a con 0x7f29e413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient(hunting): picked mon.noname-c con 0x7f29e41430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e4142520 con 0x7f29e413f710 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40b28a0 con 0x7f29e41430c0 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e40b2a30 con 0x7f29e40a98f0 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:17.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.829+0000 7f29e937b640 10 monclient(hunting): authenticate will time out at 7298.092341s 2024-03-25T18:14:17.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.840+0000 7f29e37fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f29e41430c0 0x7f29e414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41726/0 (socket says 172.21.15.132:41726) 2024-03-25T18:14:17.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.840+0000 7f29e37fe640 1 -- 172.21.15.132:0/1599292353 learned_addr learned my addr 172.21.15.132:0/1599292353 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:17.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.855+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689948906 0 0) 0x7f29e40b28a0 con 0x7f29e41430c0 2024-03-25T18:14:17.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.855+0000 7f29e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.855+0000 7f29e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.855+0000 7f29e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.855+0000 7f29e1ffb640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:14:17.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.855+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29d0003520 con 0x7f29e41430c0 2024-03-25T18:14:17.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.859+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882090940 0 0) 0x7f29e40b2a30 con 0x7f29e40a98f0 2024-03-25T18:14:17.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.859+0000 7f29e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.859+0000 7f29e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.859+0000 7f29e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.859+0000 7f29e1ffb640 10 monclient(hunting): my global_id is 4244 2024-03-25T18:14:17.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.859+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e40b28a0 con 0x7f29e40a98f0 2024-03-25T18:14:17.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.863+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638845625 0 0) 0x7f29e4142520 con 0x7f29e413f710 2024-03-25T18:14:17.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.863+0000 7f29e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.863+0000 7f29e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.863+0000 7f29e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.863+0000 7f29e1ffb640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:14:17.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.863+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e40b2a30 con 0x7f29e413f710 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3230971321 0 0) 0x7f29d0003520 con 0x7f29e41430c0 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 >> v1:172.21.15.132:6790/0 conn(0x7f29e40a98f0 legacy=0x7f29e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 >> v1:172.21.15.132:6789/0 conn(0x7f29e413f710 legacy=0x7f29e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40b44c0 con 0x7f29e41430c0 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29dc001690 con 0x7f29e41430c0 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:17.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e1ffb640 10 monclient: dump: 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e937b640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:14:17.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.871+0000 7f29e937b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:17.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.881+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29dc0018b0 con 0x7f29e41430c0 2024-03-25T18:14:17.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.881+0000 7f29e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:17.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.881+0000 7f29e1ffb640 1 -- 172.21.15.132:0/1599292353 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29dc001c40 con 0x7f29e41430c0 2024-03-25T18:14:17.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.881+0000 7f29e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.881+0000 7f29e1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.881+0000 7f29e1ffb640 10 monclient: dump: 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:17.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 4 set_mon_vals no callback set 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 10 monclient: shutdown 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 1 -- 172.21.15.132:0/1599292353 >> v1:172.21.15.161:6789/0 conn(0x7f29e41430c0 legacy=0x7f29e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 1 -- 172.21.15.132:0/1599292353 shutdown_connections 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 1 -- 172.21.15.132:0/1599292353 >> 172.21.15.132:0/1599292353 conn(0x7f29e40b7ea0 msgr2=0x7f29e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 1 -- 172.21.15.132:0/1599292353 shutdown_connections 2024-03-25T18:14:17.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.882+0000 7f29e937b640 1 -- 172.21.15.132:0/1599292353 wait complete. 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.883+0000 7f29e937b640 10 monclient: build_initial_monmap 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.883+0000 7f29e937b640 10 monclient: monmap: 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:17.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.883+0000 7f29e937b640 1 librados: starting msgr at 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.883+0000 7f29e937b640 1 librados: starting objecter 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 Processor -- start 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 -- start start 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 librados: setting wanted keys 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 librados: calling monclient init 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient: init 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient(hunting): picked mon.noname-b con 0x7f29e40a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient(hunting): picked mon.noname-c con 0x7f29e413f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient(hunting): picked mon.noname-a con 0x7f29e41430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e4135230 con 0x7f29e41430c0 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e4135da0 con 0x7f29e40a98f0 2024-03-25T18:14:17.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29e4135f30 con 0x7f29e413f710 2024-03-25T18:14:17.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:17.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.884+0000 7f29e937b640 10 monclient(hunting): authenticate will time out at 7298.147495s 2024-03-25T18:14:17.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.891+0000 7f29e2ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f29e40a98f0 0x7f29e41d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41764/0 (socket says 172.21.15.132:41764) 2024-03-25T18:14:17.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.891+0000 7f29e2ffd640 1 -- 172.21.15.132:0/2413591304 learned_addr learned my addr 172.21.15.132:0/2413591304 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:17.898 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.897+0000 7f29db7fe640 10 client.?.objecter ms_handle_connect 0x7f29e413f710 2024-03-25T18:14:17.898 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.897+0000 7f29db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.901+0000 7f29db7fe640 10 client.?.objecter ms_handle_connect 0x7f29e40a98f0 2024-03-25T18:14:17.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.901+0000 7f29db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.905+0000 7f29db7fe640 10 client.?.objecter ms_handle_connect 0x7f29e41430c0 2024-03-25T18:14:17.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.905+0000 7f29db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:17.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.911+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351668407 0 0) 0x7f29e4135da0 con 0x7f29e40a98f0 2024-03-25T18:14:17.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.911+0000 7f29db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.911+0000 7f29db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.911+0000 7f29db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.911+0000 7f29db7fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:14:17.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.911+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29c4004eb0 con 0x7f29e40a98f0 2024-03-25T18:14:17.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.914+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802446135 0 0) 0x7f29e4135230 con 0x7f29e41430c0 2024-03-25T18:14:17.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.914+0000 7f29db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.914+0000 7f29db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.914+0000 7f29db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.914+0000 7f29db7fe640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:14:17.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.914+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e4135da0 con 0x7f29e41430c0 2024-03-25T18:14:17.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.918+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232524929 0 0) 0x7f29e4135f30 con 0x7f29e413f710 2024-03-25T18:14:17.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.918+0000 7f29db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:17.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.918+0000 7f29db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:17.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.918+0000 7f29db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:17.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.918+0000 7f29db7fe640 10 monclient(hunting): my global_id is 4247 2024-03-25T18:14:17.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.918+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29e4135230 con 0x7f29e413f710 2024-03-25T18:14:17.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 163400177 0 0) 0x7f29c4004eb0 con 0x7f29e40a98f0 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29e4135f30 con 0x7f29e40a98f0 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29d4001db0 con 0x7f29e40a98f0 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:17.928 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:17.929 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:17.929 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:17.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.927+0000 7f29db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.935+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4239776785 0 0) 0x7f29e4135da0 con 0x7f29e41430c0 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.935+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29c4004eb0 con 0x7f29e41430c0 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.935+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29dc001a20 con 0x7f29e41430c0 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.935+0000 7f29db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.935+0000 7f29db7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.935+0000 7f29db7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:17.936 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.937 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:17.937 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:17.937 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:17.937 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:17.937 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3119159056 0 0) 0x7f29c4004eb0 con 0x7f29e41430c0 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 >> v1:172.21.15.132:6790/0 conn(0x7f29e413f710 legacy=0x7f29e41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 >> v1:172.21.15.161:6789/0 conn(0x7f29e40a98f0 legacy=0x7f29e41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 1 monclient: found mon.a 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e41360c0 con 0x7f29e41430c0 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 10 monclient: _renew_subs 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e41d5000 con 0x7f29e41430c0 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 10 monclient: _renew_subs 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e41d41e0 con 0x7f29e41430c0 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.941+0000 7f29e937b640 1 librados: init done 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.942+0000 7f29e937b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.942+0000 7f29e937b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.942+0000 7f29e937b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:17.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:17.942+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e4135090 con 0x7f29e41430c0 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29dc0047e0 con 0x7f29e41430c0 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29dc004950 con 0x7f29e41430c0 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29db7fe640 10 monclient: dump: 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:18.235 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.234+0000 7f29e17fa640 4 set_mon_vals no callback set 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.235+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f29dc01a920 con 0x7f29e41430c0 2024-03-25T18:14:18.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.235+0000 7f29e2ffd640 10 monclient: get_auth_request con 0x7f29c406dfb0 auth_method 0 2024-03-25T18:14:18.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.236+0000 7f29db7fe640 10 client.4254.objecter ms_handle_connect 0x7f29c406dfb0 2024-03-25T18:14:18.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f29dc054530 con 0x7f29e41430c0 2024-03-25T18:14:18.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 10 client.4254.objecter ms_dispatch 0x7f29e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:18.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:18.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:18.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-25T18:14:18.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29dc04fe70 con 0x7f29e41430c0 2024-03-25T18:14:18.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:18.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.240+0000 7f29db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:18.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.469+0000 7f29e937b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:18.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.469+0000 7f29e937b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:18.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.469+0000 7f29e937b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:18.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.469+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29e413b6e0 con 0x7f29e41430c0 2024-03-25T18:14:18.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.509+0000 7f29db7fe640 1 -- 172.21.15.132:0/2413591304 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f29dc051f70 con 0x7f29e41430c0 2024-03-25T18:14:18.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.509+0000 7f29db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:18.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.509+0000 7f29db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:18.510 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:18.510 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:18.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 10 librados: watch_flush enter 2024-03-25T18:14:18.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 10 librados: watch_flush exit 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 10 client.4254.objecter successfully canceled tick 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 >> v1:172.21.15.161:6800/411950978 conn(0x7f29c406dfb0 legacy=0x7f29c4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 10 monclient: shutdown 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 >> v1:172.21.15.132:6789/0 conn(0x7f29e41430c0 legacy=0x7f29e41db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 shutdown_connections 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 >> 172.21.15.132:0/2413591304 conn(0x7f29e40b7ea0 msgr2=0x7f29e414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 shutdown_connections 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 -- 172.21.15.132:0/2413591304 wait complete. 2024-03-25T18:14:18.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:18.512+0000 7f29e937b640 1 librados: shutdown 2024-03-25T18:14:18.525 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:25.332 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.446+0000 7f06f6206640 10 monclient: get_monmap_and_config 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.446+0000 7f06f6206640 10 monclient: build_initial_monmap 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.446+0000 7f06f6206640 10 monclient: monmap: 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:25.447385+0000 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:25.447385+0000 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:25.447 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:25.448 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:25.448 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:25.448 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.447+0000 7f06f6206640 1 Processor -- start 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.447+0000 7f06f6206640 1 -- start start 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.447+0000 7f06f6206640 10 monclient: init 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient(hunting): picked mon.noname-b con 0x7f06f00a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:25.449 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient(hunting): picked mon.noname-a con 0x7f06f013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:25.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient(hunting): picked mon.noname-c con 0x7f06f01430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:25.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f0142520 con 0x7f06f013f710 2024-03-25T18:14:25.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f00b28a0 con 0x7f06f01430c0 2024-03-25T18:14:25.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f00b2a30 con 0x7f06f00a98f0 2024-03-25T18:14:25.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:25.450 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.448+0000 7f06f6206640 10 monclient(hunting): authenticate will time out at 7305.711015s 2024-03-25T18:14:25.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.461+0000 7f06ef7fe640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f06f00a98f0 0x7f06f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:49572/0 (socket says 172.21.15.132:49572) 2024-03-25T18:14:25.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.461+0000 7f06ef7fe640 1 -- 172.21.15.132:0/2290333950 learned_addr learned my addr 172.21.15.132:0/2290333950 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:25.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.469+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178642861 0 0) 0x7f06f00b2a30 con 0x7f06f00a98f0 2024-03-25T18:14:25.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.469+0000 7f06ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.469+0000 7f06ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.469+0000 7f06ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.469+0000 7f06ee7fc640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:14:25.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.469+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d8003520 con 0x7f06f00a98f0 2024-03-25T18:14:25.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.471+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318715558 0 0) 0x7f06f00b28a0 con 0x7f06f01430c0 2024-03-25T18:14:25.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.471+0000 7f06ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.471+0000 7f06ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.471+0000 7f06ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.471+0000 7f06ee7fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:14:25.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.471+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f00b2a30 con 0x7f06f01430c0 2024-03-25T18:14:25.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.482+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318394558 0 0) 0x7f06f0142520 con 0x7f06f013f710 2024-03-25T18:14:25.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.482+0000 7f06ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.482+0000 7f06ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.482+0000 7f06ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.482+0000 7f06ee7fc640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:14:25.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.482+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f00b28a0 con 0x7f06f013f710 2024-03-25T18:14:25.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927957801 0 0) 0x7f06f00b2a30 con 0x7f06f01430c0 2024-03-25T18:14:25.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 >> v1:172.21.15.132:6790/0 conn(0x7f06f00a98f0 legacy=0x7f06f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 >> v1:172.21.15.132:6789/0 conn(0x7f06f013f710 legacy=0x7f06f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f00b44c0 con 0x7f06f01430c0 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f06e80019f0 con 0x7f06f01430c0 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:25.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06ee7fc640 10 monclient: dump: 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06f6206640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:14:25.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.491+0000 7f06f6206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06e8001b50 con 0x7f06f01430c0 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06ee7fc640 1 -- 172.21.15.132:0/2290333950 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f06e8001f20 con 0x7f06f01430c0 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06ee7fc640 10 monclient: dump: 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:25.497 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06f6206640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:25.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.496+0000 7f06f6206640 4 set_mon_vals no callback set 2024-03-25T18:14:25.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 10 monclient: shutdown 2024-03-25T18:14:25.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:25.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 1 -- 172.21.15.132:0/2290333950 >> v1:172.21.15.161:6789/0 conn(0x7f06f01430c0 legacy=0x7f06f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 1 -- 172.21.15.132:0/2290333950 shutdown_connections 2024-03-25T18:14:25.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 1 -- 172.21.15.132:0/2290333950 >> 172.21.15.132:0/2290333950 conn(0x7f06f00b7ea0 msgr2=0x7f06f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 1 -- 172.21.15.132:0/2290333950 shutdown_connections 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.497+0000 7f06f6206640 1 -- 172.21.15.132:0/2290333950 wait complete. 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 10 monclient: build_initial_monmap 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 10 monclient: monmap: 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 1 librados: starting msgr at 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 1 librados: starting objecter 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 1 Processor -- start 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 1 -- start start 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 1 librados: setting wanted keys 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 1 librados: calling monclient init 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.498+0000 7f06f6206640 10 monclient: init 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient(hunting): picked mon.noname-a con 0x7f06f00a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient(hunting): picked mon.noname-b con 0x7f06f013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient(hunting): picked mon.noname-c con 0x7f06f01430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f0139090 con 0x7f06f00a98f0 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f01db010 con 0x7f06f013f710 2024-03-25T18:14:25.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f01dac50 con 0x7f06f01430c0 2024-03-25T18:14:25.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:25.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.499+0000 7f06f6206640 10 monclient(hunting): authenticate will time out at 7305.762380s 2024-03-25T18:14:25.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.503+0000 7f06eeffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f06f013f710 0x7f06f01db530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:37270/0 (socket says 172.21.15.132:37270) 2024-03-25T18:14:25.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.503+0000 7f06eeffd640 1 -- 172.21.15.132:0/4202962993 learned_addr learned my addr 172.21.15.132:0/4202962993 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:25.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.504+0000 7f06d7fff640 10 client.?.objecter ms_handle_connect 0x7f06f013f710 2024-03-25T18:14:25.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.504+0000 7f06d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:25.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.510+0000 7f06d7fff640 10 client.?.objecter ms_handle_connect 0x7f06f01430c0 2024-03-25T18:14:25.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.510+0000 7f06d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:25.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.515+0000 7f06d7fff640 10 client.?.objecter ms_handle_connect 0x7f06f00a98f0 2024-03-25T18:14:25.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.515+0000 7f06d7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:25.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.519+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441889131 0 0) 0x7f06f01db010 con 0x7f06f013f710 2024-03-25T18:14:25.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.519+0000 7f06d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.519+0000 7f06d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.519+0000 7f06d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.519+0000 7f06d7fff640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:14:25.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.519+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06dc004eb0 con 0x7f06f013f710 2024-03-25T18:14:25.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.521+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213547723 0 0) 0x7f06f01dac50 con 0x7f06f01430c0 2024-03-25T18:14:25.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.521+0000 7f06d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.521+0000 7f06d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.521+0000 7f06d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.521+0000 7f06d7fff640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:14:25.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.521+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f01db010 con 0x7f06f01430c0 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1105367802 0 0) 0x7f06dc004eb0 con 0x7f06f013f710 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f01dac50 con 0x7f06f013f710 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f06e4001aa0 con 0x7f06f013f710 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 10 monclient(hunting): dump: 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:25.537 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.536+0000 7f06d7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:25.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.540+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215987545 0 0) 0x7f06f0139090 con 0x7f06f00a98f0 2024-03-25T18:14:25.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.540+0000 7f06d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:25.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.540+0000 7f06d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:25.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.540+0000 7f06d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:25.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.540+0000 7f06d7fff640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:14:25.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.540+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06dc004eb0 con 0x7f06f00a98f0 2024-03-25T18:14:25.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.550+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2312686959 0 0) 0x7f06dc004eb0 con 0x7f06f00a98f0 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.550+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f0139090 con 0x7f06f00a98f0 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.550+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f06e00017b0 con 0x7f06f00a98f0 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.550+0000 7f06d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.550+0000 7f06d7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.550+0000 7f06d7fff640 10 monclient(hunting): dump: 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:25.551 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.553+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033321940 0 0) 0x7f06f01dac50 con 0x7f06f013f710 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 >> v1:172.21.15.132:6790/0 conn(0x7f06f01430c0 legacy=0x7f06f01e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 >> v1:172.21.15.132:6789/0 conn(0x7f06f00a98f0 legacy=0x7f06f0139440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 1 monclient: found mon.b 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f01dade0 con 0x7f06f013f710 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:25.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 10 monclient: _renew_subs 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f00b7240 con 0x7f06f013f710 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 10 monclient: _renew_subs 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f0134f90 con 0x7f06f013f710 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.554+0000 7f06f6206640 1 librados: init done 2024-03-25T18:14:25.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.555+0000 7f06f6206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.555+0000 7f06f6206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.555+0000 7f06f6206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:25.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.555+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f014f2d0 con 0x7f06f013f710 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06e4001d00 con 0x7f06f013f710 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f06e4003a60 con 0x7f06f013f710 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06d7fff640 10 monclient: dump: 2024-03-25T18:14:25.840 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:25.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.839+0000 7f06edffb640 4 set_mon_vals no callback set 2024-03-25T18:14:25.842 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.840+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f06e4003c80 con 0x7f06f013f710 2024-03-25T18:14:25.842 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.841+0000 7f06ef7fe640 10 monclient: get_auth_request con 0x7f06dc06df70 auth_method 0 2024-03-25T18:14:25.842 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.841+0000 7f06d7fff640 10 client.4261.objecter ms_handle_connect 0x7f06dc06df70 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.848+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f06e4054900 con 0x7f06f013f710 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.848+0000 7f06d7fff640 10 client.4261.objecter ms_dispatch 0x7f06f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.848+0000 7f06d7fff640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.848+0000 7f06d7fff640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.848+0000 7f06d7fff640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.849+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06e4050240 con 0x7f06f013f710 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.849+0000 7f06d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:25.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:25.849+0000 7f06d7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:26.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.097+0000 7f06f6206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:26.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.097+0000 7f06f6206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:26.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.097+0000 7f06f6206640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:26.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.097+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f01db1a0 con 0x7f06f013f710 2024-03-25T18:14:26.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.125+0000 7f06d7fff640 1 -- 172.21.15.132:0/4202962993 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f06e4052340 con 0x7f06f013f710 2024-03-25T18:14:26.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.125+0000 7f06d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:26.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.125+0000 7f06d7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:26.126 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:26.126 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:26.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 10 librados: watch_flush enter 2024-03-25T18:14:26.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 10 librados: watch_flush exit 2024-03-25T18:14:26.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:14:26.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 >> v1:172.21.15.161:6800/411950978 conn(0x7f06dc06df70 legacy=0x7f06dc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 10 monclient: shutdown 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 >> v1:172.21.15.161:6789/0 conn(0x7f06f013f710 legacy=0x7f06f01db530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 shutdown_connections 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 >> 172.21.15.132:0/4202962993 conn(0x7f06f00b7ea0 msgr2=0x7f06f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 shutdown_connections 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 -- 172.21.15.132:0/4202962993 wait complete. 2024-03-25T18:14:26.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:26.128+0000 7f06f6206640 1 librados: shutdown 2024-03-25T18:14:26.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:29.086 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:14.350 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.168 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:14.433 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.204 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:14.461 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.256 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:14.520 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.348 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:14.589 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.407 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:14.649 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.415 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:14.679 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.501 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:14.765 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:29.627 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:14.869 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:32.948 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:33.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.063+0000 7fa8dd481640 10 monclient: get_monmap_and_config 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.063+0000 7fa8dd481640 10 monclient: build_initial_monmap 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.063+0000 7fa8dd481640 10 monclient: monmap: 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:33.064368+0000 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:33.064368+0000 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:33.064 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:33.065 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 1 Processor -- start 2024-03-25T18:14:33.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 1 -- start start 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient: init 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient(hunting): picked mon.noname-a con 0x7fa8d8061830 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient(hunting): picked mon.noname-c con 0x7fa8d81400e0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient(hunting): picked mon.noname-b con 0x7fa8d81499d0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8072600 con 0x7fa8d8061830 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8140c80 con 0x7fa8d81400e0 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8072830 con 0x7fa8d81499d0 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:33.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.064+0000 7fa8dd481640 10 monclient(hunting): authenticate will time out at 7313.327688s 2024-03-25T18:14:33.078 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.077+0000 7fa8d67fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa8d81400e0 0x7fa8d8140570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:33890/0 (socket says 172.21.15.132:33890) 2024-03-25T18:14:33.078 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.077+0000 7fa8d67fc640 1 -- 172.21.15.132:0/3033727707 learned_addr learned my addr 172.21.15.132:0/3033727707 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:33.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.091+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474882584 0 0) 0x7fa8d8140c80 con 0x7fa8d81400e0 2024-03-25T18:14:33.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.091+0000 7fa8d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.091+0000 7fa8d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.091+0000 7fa8d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.091+0000 7fa8d5ffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:14:33.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.091+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c0003520 con 0x7fa8d81400e0 2024-03-25T18:14:33.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4095958213 0 0) 0x7fa8c0003520 con 0x7fa8d81400e0 2024-03-25T18:14:33.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 >> v1:172.21.15.132:6790/0 conn(0x7fa8d81499d0 legacy=0x7fa8d814bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 >> v1:172.21.15.132:6789/0 conn(0x7fa8d8061830 legacy=0x7fa8d813d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8072b10 con 0x7fa8d81400e0 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa8cc001690 con 0x7fa8d81400e0 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8d5ffb640 10 monclient: dump: 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.104 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8dd481640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:14:33.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.102+0000 7fa8dd481640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8cc0017f0 con 0x7fa8d81400e0 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8d5ffb640 1 -- 172.21.15.132:0/3033727707 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa8cc001c00 con 0x7fa8d81400e0 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8dd481640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8dd481640 4 set_mon_vals no callback set 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:33.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.138+0000 7fa8d5ffb640 10 monclient: dump: 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:33.140 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 10 monclient: shutdown 2024-03-25T18:14:33.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 1 -- 172.21.15.132:0/3033727707 >> v1:172.21.15.161:6789/0 conn(0x7fa8d81400e0 legacy=0x7fa8d8140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 1 -- 172.21.15.132:0/3033727707 shutdown_connections 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 1 -- 172.21.15.132:0/3033727707 >> 172.21.15.132:0/3033727707 conn(0x7fa8d80b0ef0 msgr2=0x7fa8d80b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 1 -- 172.21.15.132:0/3033727707 shutdown_connections 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.139+0000 7fa8dd481640 1 -- 172.21.15.132:0/3033727707 wait complete. 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.140+0000 7fa8dd481640 10 monclient: build_initial_monmap 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.140+0000 7fa8dd481640 10 monclient: monmap: 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:33.142 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.140+0000 7fa8dd481640 1 librados: starting msgr at 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.140+0000 7fa8dd481640 1 librados: starting objecter 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 Processor -- start 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 -- start start 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 librados: setting wanted keys 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 librados: calling monclient init 2024-03-25T18:14:33.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient: init 2024-03-25T18:14:33.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:33.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:33.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient(hunting): picked mon.noname-a con 0x7fa8d8061830 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:33.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient(hunting): picked mon.noname-c con 0x7fa8d81400e0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:33.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient(hunting): picked mon.noname-b con 0x7fa8d81499d0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:33.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d81d5450 con 0x7fa8d8061830 2024-03-25T18:14:33.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d80ab230 con 0x7fa8d81499d0 2024-03-25T18:14:33.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d81d62d0 con 0x7fa8d81400e0 2024-03-25T18:14:33.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:33.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.141+0000 7fa8dd481640 10 monclient(hunting): authenticate will time out at 7313.404625s 2024-03-25T18:14:33.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.158+0000 7fa8d6ffd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa8d8061830 0x7fa8d8136bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:59242/0 (socket says 172.21.15.132:59242) 2024-03-25T18:14:33.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.158+0000 7fa8d6ffd640 1 -- 172.21.15.132:0/2503224044 learned_addr learned my addr 172.21.15.132:0/2503224044 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:33.163 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.162+0000 7fa8bf7fe640 10 client.?.objecter ms_handle_connect 0x7fa8d81499d0 2024-03-25T18:14:33.163 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.162+0000 7fa8bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:33.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.167+0000 7fa8bf7fe640 10 client.?.objecter ms_handle_connect 0x7fa8d8061830 2024-03-25T18:14:33.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.167+0000 7fa8bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:33.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.169+0000 7fa8bf7fe640 10 client.?.objecter ms_handle_connect 0x7fa8d81400e0 2024-03-25T18:14:33.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.169+0000 7fa8bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:33.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.179+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278796458 0 0) 0x7fa8d81d5450 con 0x7fa8d8061830 2024-03-25T18:14:33.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.179+0000 7fa8bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.179+0000 7fa8bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.179+0000 7fa8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.179+0000 7fa8bf7fe640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:14:33.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.179+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8c4004eb0 con 0x7fa8d8061830 2024-03-25T18:14:33.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.180+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790464338 0 0) 0x7fa8d80ab230 con 0x7fa8d81499d0 2024-03-25T18:14:33.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.180+0000 7fa8bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.180+0000 7fa8bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.180+0000 7fa8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.180+0000 7fa8bf7fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:14:33.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.180+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d81d5450 con 0x7fa8d81499d0 2024-03-25T18:14:33.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.183+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930237603 0 0) 0x7fa8d81d62d0 con 0x7fa8d81400e0 2024-03-25T18:14:33.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.183+0000 7fa8bf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.183+0000 7fa8bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.183+0000 7fa8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.183+0000 7fa8bf7fe640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:14:33.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.183+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d80ab230 con 0x7fa8d81400e0 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821400681 0 0) 0x7fa8c4004eb0 con 0x7fa8d8061830 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8d81d62d0 con 0x7fa8d8061830 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa8c8001a30 con 0x7fa8d8061830 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:33.197 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.196+0000 7fa8bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.198+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4094608627 0 0) 0x7fa8d81d5450 con 0x7fa8d81499d0 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.198+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8c4004eb0 con 0x7fa8d81499d0 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.198+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa8d0001b00 con 0x7fa8d81499d0 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.198+0000 7fa8bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.198+0000 7fa8bf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.198+0000 7fa8bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:33.199 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:33.200 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.204+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1510555282 0 0) 0x7fa8d80ab230 con 0x7fa8d81400e0 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.204+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8d81d5450 con 0x7fa8d81400e0 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.204+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa8cc001b00 con 0x7fa8d81400e0 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.204+0000 7fa8bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.204+0000 7fa8bf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.204+0000 7fa8bf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:33.205 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1659820243 0 0) 0x7fa8d81d62d0 con 0x7fa8d8061830 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 >> v1:172.21.15.132:6790/0 conn(0x7fa8d81400e0 legacy=0x7fa8d81d5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 >> v1:172.21.15.161:6789/0 conn(0x7fa8d81499d0 legacy=0x7fa8d8133490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 1 monclient: found mon.a 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d81d56a0 con 0x7fa8d8061830 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8dd481640 5 monclient: authenticate success, global_id 4266 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8dd481640 10 monclient: _renew_subs 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.214+0000 7fa8dd481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:33.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.215+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d8134250 con 0x7fa8d8061830 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.215+0000 7fa8dd481640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.215+0000 7fa8dd481640 10 monclient: _renew_subs 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.215+0000 7fa8dd481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.215+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d8133ba0 con 0x7fa8d8061830 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.215+0000 7fa8dd481640 1 librados: init done 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.216+0000 7fa8dd481640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.216+0000 7fa8dd481640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.216+0000 7fa8dd481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:33.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.216+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d80627b0 con 0x7fa8d8061830 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8c8001c70 con 0x7fa8d8061830 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa8c8003a60 con 0x7fa8d8061830 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8bf7fe640 10 monclient: dump: 2024-03-25T18:14:33.500 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:33.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.499+0000 7fa8d57fa640 4 set_mon_vals no callback set 2024-03-25T18:14:33.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.500+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fa8c8003c40 con 0x7fa8d8061830 2024-03-25T18:14:33.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.500+0000 7fa8d67fc640 10 monclient: get_auth_request con 0x7fa8c406e0f0 auth_method 0 2024-03-25T18:14:33.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.501+0000 7fa8bf7fe640 10 client.4266.objecter ms_handle_connect 0x7fa8c406e0f0 2024-03-25T18:14:33.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.506+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fa8c8054890 con 0x7fa8d8061830 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.506+0000 7fa8bf7fe640 10 client.4266.objecter ms_dispatch 0x7fa8d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.506+0000 7fa8bf7fe640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.506+0000 7fa8bf7fe640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.507+0000 7fa8bf7fe640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.507+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8c80503a0 con 0x7fa8d8061830 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.507+0000 7fa8bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.507+0000 7fa8bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:33.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.742+0000 7fa8dd481640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:33.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.742+0000 7fa8dd481640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:33.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.742+0000 7fa8dd481640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:33.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.742+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8d81d5830 con 0x7fa8d8061830 2024-03-25T18:14:33.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.769+0000 7fa8bf7fe640 1 -- 172.21.15.132:0/2503224044 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fa8c80522d0 con 0x7fa8d8061830 2024-03-25T18:14:33.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.769+0000 7fa8bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:33.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.769+0000 7fa8bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:33.771 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:33.771 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:33.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 10 librados: watch_flush enter 2024-03-25T18:14:33.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 10 librados: watch_flush exit 2024-03-25T18:14:33.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-25T18:14:33.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 10 client.4266.objecter successfully canceled tick 2024-03-25T18:14:33.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 >> v1:172.21.15.161:6800/411950978 conn(0x7fa8c406e0f0 legacy=0x7fa8c40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 10 monclient: shutdown 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 >> v1:172.21.15.132:6789/0 conn(0x7fa8d8061830 legacy=0x7fa8d8136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 shutdown_connections 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 >> 172.21.15.132:0/2503224044 conn(0x7fa8d80b0ef0 msgr2=0x7fa8d80afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 shutdown_connections 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 -- 172.21.15.132:0/2503224044 wait complete. 2024-03-25T18:14:33.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:33.772+0000 7fa8dd481640 1 librados: shutdown 2024-03-25T18:14:33.784 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:40.590 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:40.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.705+0000 7f320ff23640 10 monclient: get_monmap_and_config 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.705+0000 7f320ff23640 10 monclient: build_initial_monmap 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.705+0000 7f320ff23640 10 monclient: monmap: 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:40.706290+0000 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:40.706290+0000 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:40.706 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:40.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.706+0000 7f320ff23640 1 Processor -- start 2024-03-25T18:14:40.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.706+0000 7f320ff23640 1 -- start start 2024-03-25T18:14:40.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.706+0000 7f320ff23640 10 monclient: init 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient(hunting): picked mon.noname-a con 0x7f32080a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient(hunting): picked mon.noname-b con 0x7f320813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient(hunting): picked mon.noname-c con 0x7f32081430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3208142520 con 0x7f32080a98f0 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32080b28a0 con 0x7f32081430c0 2024-03-25T18:14:40.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32080b2a30 con 0x7f320813f710 2024-03-25T18:14:40.709 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:40.709 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.707+0000 7f320ff23640 10 monclient(hunting): authenticate will time out at 7320.969986s 2024-03-25T18:14:40.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.710+0000 7f320dcbf640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f32080a98f0 0x7f32080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:35824/0 (socket says 172.21.15.132:35824) 2024-03-25T18:14:40.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.710+0000 7f320dcbf640 1 -- 172.21.15.132:0/3465445090 learned_addr learned my addr 172.21.15.132:0/3465445090 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:40.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.725+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2606338517 0 0) 0x7f3208142520 con 0x7f32080a98f0 2024-03-25T18:14:40.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.725+0000 7f320ccbd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.725+0000 7f320ccbd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.725+0000 7f320ccbd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.725+0000 7f320ccbd640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:14:40.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.725+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f4003520 con 0x7f32080a98f0 2024-03-25T18:14:40.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.727+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070265881 0 0) 0x7f32080b28a0 con 0x7f32081430c0 2024-03-25T18:14:40.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.727+0000 7f320ccbd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.727+0000 7f320ccbd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.727+0000 7f320ccbd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.727+0000 7f320ccbd640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:14:40.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.727+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3208142520 con 0x7f32081430c0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652254293 0 0) 0x7f3208142520 con 0x7f32081430c0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 >> v1:172.21.15.132:6790/0 conn(0x7f320813f710 legacy=0x7f32081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 >> v1:172.21.15.132:6789/0 conn(0x7f32080a98f0 legacy=0x7f32080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 1 monclient: found mon.noname-c 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32080b44c0 con 0x7f32081430c0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f32000019f0 con 0x7f32081430c0 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ff23640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:14:40.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ff23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ccbd640 10 monclient: dump: 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:40.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.742+0000 7f320ff23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:40.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3200001b50 con 0x7f32081430c0 2024-03-25T18:14:40.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ccbd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:40.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ccbd640 1 -- 172.21.15.132:0/3465445090 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3200001f20 con 0x7f32081430c0 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ff23640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ff23640 4 set_mon_vals no callback set 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ccbd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ccbd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.754+0000 7f320ccbd640 10 monclient: dump: 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:40.755 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 10 monclient: shutdown 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 1 -- 172.21.15.132:0/3465445090 >> v1:172.21.15.161:6789/0 conn(0x7f32081430c0 legacy=0x7f320814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 1 -- 172.21.15.132:0/3465445090 shutdown_connections 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 1 -- 172.21.15.132:0/3465445090 >> 172.21.15.132:0/3465445090 conn(0x7f32080b7ea0 msgr2=0x7f32080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 1 -- 172.21.15.132:0/3465445090 shutdown_connections 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 1 -- 172.21.15.132:0/3465445090 wait complete. 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 10 monclient: build_initial_monmap 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.755+0000 7f320ff23640 10 monclient: monmap: 2024-03-25T18:14:40.757 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 1 librados: starting msgr at 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 1 librados: starting objecter 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 1 Processor -- start 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 1 -- start start 2024-03-25T18:14:40.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 1 librados: setting wanted keys 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 1 librados: calling monclient init 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.756+0000 7f320ff23640 10 monclient: init 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient(hunting): picked mon.noname-c con 0x7f32080a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient(hunting): picked mon.noname-a con 0x7f320813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient(hunting): picked mon.noname-b con 0x7f32081430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3208138600 con 0x7f320813f710 2024-03-25T18:14:40.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32080b1620 con 0x7f32081430c0 2024-03-25T18:14:40.760 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32081346b0 con 0x7f32080a98f0 2024-03-25T18:14:40.760 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:40.760 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.757+0000 7f320ff23640 10 monclient(hunting): authenticate will time out at 7321.020063s 2024-03-25T18:14:40.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.769+0000 7f320e4c0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f32081430c0 0x7f32081da880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:38602/0 (socket says 172.21.15.132:38602) 2024-03-25T18:14:40.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.769+0000 7f320e4c0640 1 -- 172.21.15.132:0/3723314334 learned_addr learned my addr 172.21.15.132:0/3723314334 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:40.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.772+0000 7f31fe7fc640 10 client.?.objecter ms_handle_connect 0x7f32081430c0 2024-03-25T18:14:40.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.772+0000 7f31fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.778 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.777+0000 7f31fe7fc640 10 client.?.objecter ms_handle_connect 0x7f320813f710 2024-03-25T18:14:40.778 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.777+0000 7f31fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.781 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.780+0000 7f31fe7fc640 10 client.?.objecter ms_handle_connect 0x7f32080a98f0 2024-03-25T18:14:40.781 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.780+0000 7f31fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:40.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.788+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429797656 0 0) 0x7f32080b1620 con 0x7f32081430c0 2024-03-25T18:14:40.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.788+0000 7f31fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.788+0000 7f31fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.788+0000 7f31fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.788+0000 7f31fe7fc640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:14:40.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.788+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ec004eb0 con 0x7f32081430c0 2024-03-25T18:14:40.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.795+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248440491 0 0) 0x7f32081346b0 con 0x7f32080a98f0 2024-03-25T18:14:40.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.795+0000 7f31fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.795+0000 7f31fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.795+0000 7f31fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.795+0000 7f31fe7fc640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:14:40.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.795+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32080b1620 con 0x7f32080a98f0 2024-03-25T18:14:40.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.802+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539486517 0 0) 0x7f3208138600 con 0x7f320813f710 2024-03-25T18:14:40.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.802+0000 7f31fe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:40.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.802+0000 7f31fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:40.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.802+0000 7f31fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:40.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.802+0000 7f31fe7fc640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:14:40.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.802+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32081346b0 con 0x7f320813f710 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 612459778 0 0) 0x7f31ec004eb0 con 0x7f32081430c0 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3208138600 con 0x7f32081430c0 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3200001f60 con 0x7f32081430c0 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:40.805 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:40.806 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:40.806 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:40.806 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:40.806 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:40.806 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:40.806 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.804+0000 7f31fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:40.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4055986041 0 0) 0x7f3208138600 con 0x7f32081430c0 2024-03-25T18:14:40.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 >> v1:172.21.15.132:6790/0 conn(0x7f32080a98f0 legacy=0x7f3208137690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 >> v1:172.21.15.132:6789/0 conn(0x7f320813f710 legacy=0x7f3208133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:40.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 1 monclient: found mon.b 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3208134840 con 0x7f32081430c0 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f31fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 10 monclient: _renew_subs 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3208137f50 con 0x7f32081430c0 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 10 monclient: _renew_subs 2024-03-25T18:14:40.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:40.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32081daf90 con 0x7f32081430c0 2024-03-25T18:14:40.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.815+0000 7f320ff23640 1 librados: init done 2024-03-25T18:14:40.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.816+0000 7f320ff23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.816+0000 7f320ff23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:40.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.816+0000 7f320ff23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:40.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:40.816+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32080627b0 con 0x7f32081430c0 2024-03-25T18:14:41.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3200003c20 con 0x7f32081430c0 2024-03-25T18:14:41.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3200003fb0 con 0x7f32081430c0 2024-03-25T18:14:41.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fe7fc640 10 monclient: dump: 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:41.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.099+0000 7f31fffff640 4 set_mon_vals no callback set 2024-03-25T18:14:41.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.100+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f3200004190 con 0x7f32081430c0 2024-03-25T18:14:41.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.101+0000 7f320dcbf640 10 monclient: get_auth_request con 0x7f31ec06ddf0 auth_method 0 2024-03-25T18:14:41.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.101+0000 7f31fe7fc640 10 client.4273.objecter ms_handle_connect 0x7f31ec06ddf0 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.105+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f3200054c00 con 0x7f32081430c0 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.105+0000 7f31fe7fc640 10 client.4273.objecter ms_dispatch 0x7f3208000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.105+0000 7f31fe7fc640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.105+0000 7f31fe7fc640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.105+0000 7f31fe7fc640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.106+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3200050710 con 0x7f32081430c0 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.106+0000 7f31fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.106+0000 7f31fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:41.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.341+0000 7f320ff23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.341+0000 7f320ff23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.341+0000 7f320ff23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:14:41.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.341+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32081387b0 con 0x7f32081430c0 2024-03-25T18:14:41.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.372+0000 7f31fe7fc640 1 -- 172.21.15.132:0/3723314334 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f3200052640 con 0x7f32081430c0 2024-03-25T18:14:41.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.372+0000 7f31fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.372+0000 7f31fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:41.374 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:41.374 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:41.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 10 librados: watch_flush enter 2024-03-25T18:14:41.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 10 librados: watch_flush exit 2024-03-25T18:14:41.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 10 client.4273.objecter successfully canceled tick 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 >> v1:172.21.15.161:6800/411950978 conn(0x7f31ec06ddf0 legacy=0x7f31ec0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 10 monclient: shutdown 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 >> v1:172.21.15.161:6789/0 conn(0x7f32081430c0 legacy=0x7f32081da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 shutdown_connections 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 >> 172.21.15.132:0/3723314334 conn(0x7f32080b7ea0 msgr2=0x7f3208074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 shutdown_connections 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 -- 172.21.15.132:0/3723314334 wait complete. 2024-03-25T18:14:41.377 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:41.375+0000 7f320ff23640 1 librados: shutdown 2024-03-25T18:14:41.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:48.195 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.309+0000 7f5c8657b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.309+0000 7f5c8657b640 10 monclient: build_initial_monmap 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.309+0000 7f5c8657b640 10 monclient: monmap: 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:48.310371+0000 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:48.310371+0000 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:48.310 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:48.311 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 1 Processor -- start 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 1 -- start start 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient: init 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c800a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c801430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c80142520 con 0x7f5c800a98f0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c800b28a0 con 0x7f5c8013f710 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c800b2a30 con 0x7f5c801430c0 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.310+0000 7f5c8657b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:48.312 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.311+0000 7f5c8657b640 10 monclient(hunting): authenticate will time out at 7328.573719s 2024-03-25T18:14:48.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.326+0000 7f5c7ffff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f5c800a98f0 0x7f5c800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:35648/0 (socket says 172.21.15.132:35648) 2024-03-25T18:14:48.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.326+0000 7f5c7ffff640 1 -- 172.21.15.132:0/1194860044 learned_addr learned my addr 172.21.15.132:0/1194860044 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.344+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813377453 0 0) 0x7f5c80142520 con 0x7f5c800a98f0 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.344+0000 7f5c7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.344+0000 7f5c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.344+0000 7f5c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.344+0000 7f5c7effd640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:14:48.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.344+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c68003520 con 0x7f5c800a98f0 2024-03-25T18:14:48.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.348+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030677822 0 0) 0x7f5c800b28a0 con 0x7f5c8013f710 2024-03-25T18:14:48.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.348+0000 7f5c7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.348+0000 7f5c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.348+0000 7f5c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.348+0000 7f5c7effd640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:14:48.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.348+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c80142520 con 0x7f5c8013f710 2024-03-25T18:14:48.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.350+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795973358 0 0) 0x7f5c800b2a30 con 0x7f5c801430c0 2024-03-25T18:14:48.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.350+0000 7f5c7effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.350+0000 7f5c7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.350+0000 7f5c7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.350+0000 7f5c7effd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:14:48.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.350+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c800b28a0 con 0x7f5c801430c0 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2822791795 0 0) 0x7f5c68003520 con 0x7f5c800a98f0 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 >> v1:172.21.15.132:6790/0 conn(0x7f5c801430c0 legacy=0x7f5c8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 >> v1:172.21.15.161:6789/0 conn(0x7f5c8013f710 legacy=0x7f5c801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 1 monclient: found mon.noname-a 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:14:48.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c800b44c0 con 0x7f5c800a98f0 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: _finish_auth 0 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5c70001690 con 0x7f5c800a98f0 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c7effd640 10 monclient: dump: 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:48.362 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c8657b640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:14:48.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.360+0000 7f5c8657b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c700017f0 con 0x7f5c800a98f0 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c7effd640 1 -- 172.21.15.132:0/1194860044 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5c70001c00 con 0x7f5c800a98f0 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c7effd640 10 monclient: dump: 2024-03-25T18:14:48.365 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c8657b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:48.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.364+0000 7f5c8657b640 4 set_mon_vals no callback set 2024-03-25T18:14:48.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.365+0000 7f5c8657b640 10 monclient: shutdown 2024-03-25T18:14:48.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.365+0000 7f5c8657b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:48.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.365+0000 7f5c8657b640 1 -- 172.21.15.132:0/1194860044 >> v1:172.21.15.132:6789/0 conn(0x7f5c800a98f0 legacy=0x7f5c800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.365+0000 7f5c8657b640 1 -- 172.21.15.132:0/1194860044 shutdown_connections 2024-03-25T18:14:48.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.365+0000 7f5c8657b640 1 -- 172.21.15.132:0/1194860044 >> 172.21.15.132:0/1194860044 conn(0x7f5c800b7ea0 msgr2=0x7f5c800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.365+0000 7f5c8657b640 1 -- 172.21.15.132:0/1194860044 shutdown_connections 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.366+0000 7f5c8657b640 1 -- 172.21.15.132:0/1194860044 wait complete. 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.366+0000 7f5c8657b640 10 monclient: build_initial_monmap 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.366+0000 7f5c8657b640 10 monclient: monmap: 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 1 librados: starting msgr at 2024-03-25T18:14:48.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 1 librados: starting objecter 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 1 Processor -- start 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 1 -- start start 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 1 librados: setting wanted keys 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 1 librados: calling monclient init 2024-03-25T18:14:48.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.367+0000 7f5c8657b640 10 monclient: init 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient(hunting): picked mon.noname-c con 0x7f5c800a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient(hunting): picked mon.noname-b con 0x7f5c8013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient(hunting): picked mon.noname-a con 0x7f5c801430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c80138970 con 0x7f5c801430c0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c801d7810 con 0x7f5c8013f710 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c80139990 con 0x7f5c800a98f0 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:48.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.368+0000 7f5c8657b640 10 monclient(hunting): authenticate will time out at 7328.630935s 2024-03-25T18:14:48.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.373+0000 7f5c7ffff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f5c800a98f0 0x7f5c80138260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:44156/0 (socket says 172.21.15.132:44156) 2024-03-25T18:14:48.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.373+0000 7f5c7ffff640 1 -- 172.21.15.132:0/2031770937 learned_addr learned my addr 172.21.15.132:0/2031770937 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:48.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.381+0000 7f5c7cff9640 10 client.?.objecter ms_handle_connect 0x7f5c801430c0 2024-03-25T18:14:48.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.381+0000 7f5c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.386+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040084744 0 0) 0x7f5c80138970 con 0x7f5c801430c0 2024-03-25T18:14:48.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.386+0000 7f5c7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.386+0000 7f5c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.386+0000 7f5c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.386+0000 7f5c7cff9640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:14:48.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.386+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c004eb0 con 0x7f5c801430c0 2024-03-25T18:14:48.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.387+0000 7f5c7cff9640 10 client.?.objecter ms_handle_connect 0x7f5c800a98f0 2024-03-25T18:14:48.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.387+0000 7f5c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.390+0000 7f5c7cff9640 10 client.?.objecter ms_handle_connect 0x7f5c8013f710 2024-03-25T18:14:48.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.390+0000 7f5c7cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 270792457 0 0) 0x7f5c6c004eb0 con 0x7f5c801430c0 2024-03-25T18:14:48.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c80138970 con 0x7f5c801430c0 2024-03-25T18:14:48.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5c78001bd0 con 0x7f5c801430c0 2024-03-25T18:14:48.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 10 monclient(hunting): dump: 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:48.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.392+0000 7f5c7cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:48.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.399+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630726768 0 0) 0x7f5c801d7810 con 0x7f5c8013f710 2024-03-25T18:14:48.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.399+0000 7f5c7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.399+0000 7f5c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.399+0000 7f5c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.399+0000 7f5c7cff9640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:14:48.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.399+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c6c004eb0 con 0x7f5c8013f710 2024-03-25T18:14:48.401 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.401+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540294295 0 0) 0x7f5c80139990 con 0x7f5c800a98f0 2024-03-25T18:14:48.401 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.401+0000 7f5c7cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.402 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.401+0000 7f5c7cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.402 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.401+0000 7f5c7cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.402 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.401+0000 7f5c7cff9640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:14:48.402 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.401+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c801d7810 con 0x7f5c800a98f0 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2312283495 0 0) 0x7f5c80138970 con 0x7f5c801430c0 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 >> v1:172.21.15.132:6790/0 conn(0x7f5c800a98f0 legacy=0x7f5c80138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 >> v1:172.21.15.161:6789/0 conn(0x7f5c8013f710 legacy=0x7f5c801d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 1 monclient: found mon.a 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c801d8440 con 0x7f5c801430c0 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c7cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:14:48.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 10 monclient: _renew_subs 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c801d41e0 con 0x7f5c801430c0 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 10 monclient: _renew_subs 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c80138d80 con 0x7f5c801430c0 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.412+0000 7f5c8657b640 1 librados: init done 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.413+0000 7f5c8657b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.413+0000 7f5c8657b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.413+0000 7f5c8657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:48.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.413+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c801d79a0 con 0x7f5c801430c0 2024-03-25T18:14:48.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c780020a0 con 0x7f5c801430c0 2024-03-25T18:14:48.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:48.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5c78003b30 con 0x7f5c801430c0 2024-03-25T18:14:48.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7cff9640 10 monclient: dump: 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:48.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.715+0000 7f5c7e7fc640 4 set_mon_vals no callback set 2024-03-25T18:14:48.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.717+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f5c78003d50 con 0x7f5c801430c0 2024-03-25T18:14:48.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.717+0000 7f5c7ffff640 10 monclient: get_auth_request con 0x7f5c6c06de10 auth_method 0 2024-03-25T18:14:48.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.717+0000 7f5c7cff9640 10 client.4278.objecter ms_handle_connect 0x7f5c6c06de10 2024-03-25T18:14:48.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f5c78053930 con 0x7f5c801430c0 2024-03-25T18:14:48.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 10 client.4278.objecter ms_dispatch 0x7f5c80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:48.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:48.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:48.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-25T18:14:48.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c7804f270 con 0x7f5c801430c0 2024-03-25T18:14:48.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.723+0000 7f5c7cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:48.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.966+0000 7f5c8657b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.966+0000 7f5c8657b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.966+0000 7f5c8657b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:48.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.966+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c800627b0 con 0x7f5c801430c0 2024-03-25T18:14:48.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.990+0000 7f5c7cff9640 1 -- 172.21.15.132:0/2031770937 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f5c78051370 con 0x7f5c801430c0 2024-03-25T18:14:48.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.990+0000 7f5c7cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:48.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.990+0000 7f5c7cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:48.991 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:48.991 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:48.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 10 librados: watch_flush enter 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 10 librados: watch_flush exit 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 10 client.4278.objecter successfully canceled tick 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 >> v1:172.21.15.161:6800/411950978 conn(0x7f5c6c06de10 legacy=0x7f5c6c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 10 monclient: shutdown 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 >> v1:172.21.15.132:6789/0 conn(0x7f5c801430c0 legacy=0x7f5c801d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 shutdown_connections 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.993+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 >> 172.21.15.132:0/2031770937 conn(0x7f5c800b7ea0 msgr2=0x7f5c8013c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.994+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 shutdown_connections 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.994+0000 7f5c8657b640 1 -- 172.21.15.132:0/2031770937 wait complete. 2024-03-25T18:14:48.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:48.994+0000 7f5c8657b640 1 librados: shutdown 2024-03-25T18:14:49.007 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:49.072 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:34.337 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.120 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:34.385 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.159 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:34.423 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.171 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:34.413 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.175 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:34.439 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.178 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:34.420 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.181 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:34.446 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.231 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:34.496 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:49.239 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:34.480 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:55.812 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.923+0000 7f33c68e2640 10 monclient: get_monmap_and_config 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.923+0000 7f33c68e2640 10 monclient: build_initial_monmap 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.923+0000 7f33c68e2640 10 monclient: monmap: 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:14:55.924322+0000 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:14:55.924322+0000 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:14:55.924 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:14:55.925 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:55.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 1 Processor -- start 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 1 -- start start 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient: init 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient(hunting): picked mon.noname-b con 0x7f33c00b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient(hunting): picked mon.noname-a con 0x7f33c00b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient(hunting): picked mon.noname-c con 0x7f33c014be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00af940 con 0x7f33c00b1730 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00b0a70 con 0x7f33c014be90 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00b0d50 con 0x7f33c00b4d60 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:55.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.924+0000 7f33c68e2640 10 monclient(hunting): authenticate will time out at 7336.187700s 2024-03-25T18:14:55.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.929+0000 7f33c4e7f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f33c014be90 0x7f33c014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:34836/0 (socket says 172.21.15.132:34836) 2024-03-25T18:14:55.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.929+0000 7f33c4e7f640 1 -- 172.21.15.132:0/3577777329 learned_addr learned my addr 172.21.15.132:0/3577777329 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:55.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.939+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122753152 0 0) 0x7f33c00b0a70 con 0x7f33c014be90 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.939+0000 7f33beffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.939+0000 7f33beffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.939+0000 7f33beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.939+0000 7f33beffd640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:14:55.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.939+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a8003520 con 0x7f33c014be90 2024-03-25T18:14:55.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1144178649 0 0) 0x7f33a8003520 con 0x7f33c014be90 2024-03-25T18:14:55.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 >> v1:172.21.15.132:6790/0 conn(0x7f33c00b4d60 legacy=0x7f33c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 >> v1:172.21.15.132:6789/0 conn(0x7f33c00b1730 legacy=0x7f33c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 1 monclient: found mon.noname-c 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00b1030 con 0x7f33c014be90 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f33b8001a10 con 0x7f33c014be90 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.947+0000 7f33beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.948+0000 7f33beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.948+0000 7f33beffd640 10 monclient: dump: 2024-03-25T18:14:55.949 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.948+0000 7f33c68e2640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:14:55.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.948+0000 7f33c68e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33b8001b70 con 0x7f33c014be90 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33beffd640 1 -- 172.21.15.132:0/3577777329 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f33b8001fc0 con 0x7f33c014be90 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:55.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33beffd640 10 monclient: dump: 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33c68e2640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:55.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.965+0000 7f33c68e2640 4 set_mon_vals no callback set 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 10 monclient: shutdown 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 1 -- 172.21.15.132:0/3577777329 >> v1:172.21.15.161:6789/0 conn(0x7f33c014be90 legacy=0x7f33c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 1 -- 172.21.15.132:0/3577777329 shutdown_connections 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 1 -- 172.21.15.132:0/3577777329 >> 172.21.15.132:0/3577777329 conn(0x7f33c00617a0 msgr2=0x7f33c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 1 -- 172.21.15.132:0/3577777329 shutdown_connections 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.966+0000 7f33c68e2640 1 -- 172.21.15.132:0/3577777329 wait complete. 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.967+0000 7f33c68e2640 10 monclient: build_initial_monmap 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.967+0000 7f33c68e2640 10 monclient: monmap: 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:14:55.969 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 1 librados: starting msgr at 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 1 librados: starting objecter 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 1 Processor -- start 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 1 -- start start 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 1 librados: setting wanted keys 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 1 librados: calling monclient init 2024-03-25T18:14:55.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.968+0000 7f33c68e2640 10 monclient: init 2024-03-25T18:14:55.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient(hunting): picked mon.noname-b con 0x7f33c00b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient(hunting): picked mon.noname-c con 0x7f33c00b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient(hunting): picked mon.noname-a con 0x7f33c014be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0142530 con 0x7f33c014be90 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0141bc0 con 0x7f33c00b1730 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c0141e30 con 0x7f33c00b4d60 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:55.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.969+0000 7f33c68e2640 10 monclient(hunting): authenticate will time out at 7336.232269s 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.976+0000 7f33c4e7f640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f33c014be90 0x7f33c01e6a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37318/0 (socket says 172.21.15.132:37318) 2024-03-25T18:14:55.977 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.976+0000 7f33c4e7f640 1 -- 172.21.15.132:0/2252534168 learned_addr learned my addr 172.21.15.132:0/2252534168 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:14:55.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.977+0000 7f33bcff9640 10 client.?.objecter ms_handle_connect 0x7f33c014be90 2024-03-25T18:14:55.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.977+0000 7f33bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.982+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910675433 0 0) 0x7f33c0142530 con 0x7f33c014be90 2024-03-25T18:14:55.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.982+0000 7f33bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.982+0000 7f33bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.982+0000 7f33bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.982+0000 7f33bcff9640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:14:55.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.983+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33ac004eb0 con 0x7f33c014be90 2024-03-25T18:14:55.987 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.986+0000 7f33bcff9640 10 client.?.objecter ms_handle_connect 0x7f33c00b1730 2024-03-25T18:14:55.987 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.986+0000 7f33bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.992+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381298177 0 0) 0x7f33c0141bc0 con 0x7f33c00b1730 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.992+0000 7f33bcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.992+0000 7f33bcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.992+0000 7f33bcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.992+0000 7f33bcff9640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:14:55.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.992+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33c0142530 con 0x7f33c00b1730 2024-03-25T18:14:55.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.994+0000 7f33bcff9640 10 client.?.objecter ms_handle_connect 0x7f33c00b4d60 2024-03-25T18:14:55.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.994+0000 7f33bcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:55.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663652975 0 0) 0x7f33ac004eb0 con 0x7f33c014be90 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33c0141bc0 con 0x7f33c014be90 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f33b8001ec0 con 0x7f33c014be90 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 10 monclient(hunting): dump: 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:55.998 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:55.999 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:55.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:55.997+0000 7f33bcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 771573616 0 0) 0x7f33c0141bc0 con 0x7f33c014be90 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 >> v1:172.21.15.132:6790/0 conn(0x7f33c00b4d60 legacy=0x7f33c01429a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 >> v1:172.21.15.161:6789/0 conn(0x7f33c00b1730 legacy=0x7f33c0138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 1 monclient: found mon.a 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:56.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c0135540 con 0x7f33c014be90 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 5 monclient: authenticate success, global_id 4284 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 10 monclient: _renew_subs 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c0135c10 con 0x7f33c014be90 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 10 monclient: _renew_subs 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c0140560 con 0x7f33c014be90 2024-03-25T18:14:56.014 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.012+0000 7f33c68e2640 1 librados: init done 2024-03-25T18:14:56.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.013+0000 7f33c68e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.013+0000 7f33c68e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.013+0000 7f33c68e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:56.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.013+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c00627b0 con 0x7f33c014be90 2024-03-25T18:14:56.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.312+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33b80020e0 con 0x7f33c014be90 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33bcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f33b8004fb0 con 0x7f33c014be90 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33bcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33bcff9640 10 monclient: dump: 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.314 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33be7fc640 4 set_mon_vals no callback set 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.313+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f33b801af80 con 0x7f33c014be90 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.314+0000 7f33bffff640 10 monclient: get_auth_request con 0x7f33ac06dd10 auth_method 0 2024-03-25T18:14:56.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.314+0000 7f33bcff9640 10 client.4284.objecter ms_handle_connect 0x7f33ac06dd10 2024-03-25T18:14:56.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.320+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f33b8054b30 con 0x7f33c014be90 2024-03-25T18:14:56.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.320+0000 7f33bcff9640 10 client.4284.objecter ms_dispatch 0x7f33c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:56.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.320+0000 7f33bcff9640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:56.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.320+0000 7f33bcff9640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:56.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.321+0000 7f33bcff9640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-25T18:14:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.321+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33b8050470 con 0x7f33c014be90 2024-03-25T18:14:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.321+0000 7f33bcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.321+0000 7f33bcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:56.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.573+0000 7f33c68e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.573+0000 7f33c68e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.573+0000 7f33c68e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:14:56.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.573+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33c00b5cf0 con 0x7f33c014be90 2024-03-25T18:14:56.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.599+0000 7f33bcff9640 1 -- 172.21.15.132:0/2252534168 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f33b8052570 con 0x7f33c014be90 2024-03-25T18:14:56.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.599+0000 7f33bcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:56.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.599+0000 7f33bcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:56.600 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:14:56.600 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 10 librados: watch_flush enter 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 10 librados: watch_flush exit 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 10 client.4284.objecter successfully canceled tick 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 >> v1:172.21.15.161:6800/411950978 conn(0x7f33ac06dd10 legacy=0x7f33ac0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 10 monclient: shutdown 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:56.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.601+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 >> v1:172.21.15.132:6789/0 conn(0x7f33c014be90 legacy=0x7f33c01e6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.602+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 shutdown_connections 2024-03-25T18:14:56.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.602+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 >> 172.21.15.132:0/2252534168 conn(0x7f33c00617a0 msgr2=0x7f33c00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:56.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.602+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 shutdown_connections 2024-03-25T18:14:56.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.602+0000 7f33c68e2640 1 -- 172.21.15.132:0/2252534168 wait complete. 2024-03-25T18:14:56.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:14:56.602+0000 7f33c68e2640 1 librados: shutdown 2024-03-25T18:14:56.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:03.421 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:03.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.537+0000 7fdb7c87b640 10 monclient: get_monmap_and_config 2024-03-25T18:15:03.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.537+0000 7fdb7c87b640 10 monclient: build_initial_monmap 2024-03-25T18:15:03.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.537+0000 7fdb7c87b640 10 monclient: monmap: 2024-03-25T18:15:03.538 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:03.538 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:03.538639+0000 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:03.538639+0000 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:03.539 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.538+0000 7fdb7c87b640 1 Processor -- start 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.538+0000 7fdb7c87b640 1 -- start start 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.538+0000 7fdb7c87b640 10 monclient: init 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient(hunting): picked mon.noname-b con 0x7fdb780a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient(hunting): picked mon.noname-a con 0x7fdb7813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient(hunting): picked mon.noname-c con 0x7fdb781430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb78142520 con 0x7fdb7813f710 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780b28a0 con 0x7fdb781430c0 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb780b2a30 con 0x7fdb780a98f0 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:03.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.539+0000 7fdb7c87b640 10 monclient(hunting): authenticate will time out at 7343.802064s 2024-03-25T18:15:03.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.548+0000 7fdb76d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdb781430c0 0x7fdb7814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:59408/0 (socket says 172.21.15.132:59408) 2024-03-25T18:15:03.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.548+0000 7fdb76d9d640 1 -- 172.21.15.132:0/2487795578 learned_addr learned my addr 172.21.15.132:0/2487795578 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:03.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.570+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075482967 0 0) 0x7fdb780b28a0 con 0x7fdb781430c0 2024-03-25T18:15:03.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.570+0000 7fdb7559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.570+0000 7fdb7559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.570+0000 7fdb7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.570+0000 7fdb7559a640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:15:03.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.570+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb60003520 con 0x7fdb781430c0 2024-03-25T18:15:03.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.576+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588325348 0 0) 0x7fdb78142520 con 0x7fdb7813f710 2024-03-25T18:15:03.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.576+0000 7fdb7559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.576+0000 7fdb7559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.576+0000 7fdb7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.576+0000 7fdb7559a640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:15:03.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.577+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb780b28a0 con 0x7fdb7813f710 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.582+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296525033 0 0) 0x7fdb780b2a30 con 0x7fdb780a98f0 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.582+0000 7fdb7559a640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.582+0000 7fdb7559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.582+0000 7fdb7559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.582+0000 7fdb7559a640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:15:03.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.582+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78142520 con 0x7fdb780a98f0 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391389017 0 0) 0x7fdb60003520 con 0x7fdb781430c0 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 >> v1:172.21.15.132:6790/0 conn(0x7fdb780a98f0 legacy=0x7fdb780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 >> v1:172.21.15.132:6789/0 conn(0x7fdb7813f710 legacy=0x7fdb781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: _finish_hunting 0 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 1 monclient: found mon.noname-c 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:15:03.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780b44c0 con 0x7fdb781430c0 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: _finish_auth 0 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: _check_auth_tickets 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdb70001a10 con 0x7fdb781430c0 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7559a640 10 monclient: dump: 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.591 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:03.592 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:03.592 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:03.592 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:03.592 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7c87b640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:15:03.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.589+0000 7fdb7c87b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:03.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb70001b70 con 0x7fdb781430c0 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7559a640 1 -- 172.21.15.132:0/2487795578 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdb70001f80 con 0x7fdb781430c0 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7c87b640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7c87b640 4 set_mon_vals no callback set 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.594+0000 7fdb7559a640 10 monclient: dump: 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:03.595 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:03.596 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:03.596 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:03.596 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 10 monclient: shutdown 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2487795578 >> v1:172.21.15.161:6789/0 conn(0x7fdb781430c0 legacy=0x7fdb7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2487795578 shutdown_connections 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2487795578 >> 172.21.15.132:0/2487795578 conn(0x7fdb780b7ea0 msgr2=0x7fdb780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2487795578 shutdown_connections 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.595+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2487795578 wait complete. 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.596+0000 7fdb7c87b640 10 monclient: build_initial_monmap 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.596+0000 7fdb7c87b640 10 monclient: monmap: 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:03.597 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:03.598 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.596+0000 7fdb7c87b640 1 librados: starting msgr at 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.596+0000 7fdb7c87b640 1 librados: starting objecter 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 1 Processor -- start 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 1 -- start start 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 1 librados: setting wanted keys 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 1 librados: calling monclient init 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 10 monclient: init 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 10 monclient(hunting): picked mon.noname-c con 0x7fdb780a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 10 monclient(hunting): picked mon.noname-b con 0x7fdb7813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:03.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 10 monclient(hunting): picked mon.noname-a con 0x7fdb781430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:03.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb78137c60 con 0x7fdb781430c0 2024-03-25T18:15:03.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.597+0000 7fdb7c87b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb78139160 con 0x7fdb7813f710 2024-03-25T18:15:03.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.598+0000 7fdb7c87b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb781380f0 con 0x7fdb780a98f0 2024-03-25T18:15:03.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.598+0000 7fdb7c87b640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:03.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.598+0000 7fdb7c87b640 10 monclient(hunting): authenticate will time out at 7343.860740s 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.607+0000 7fdb76d9d640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fdb781430c0 0x7fdb781e2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:45342/0 (socket says 172.21.15.132:45342) 2024-03-25T18:15:03.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.607+0000 7fdb76d9d640 1 -- 172.21.15.132:0/2583004187 learned_addr learned my addr 172.21.15.132:0/2583004187 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:03.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.608+0000 7fdb66ffd640 10 client.?.objecter ms_handle_connect 0x7fdb781430c0 2024-03-25T18:15:03.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.608+0000 7fdb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.615+0000 7fdb66ffd640 10 client.?.objecter ms_handle_connect 0x7fdb780a98f0 2024-03-25T18:15:03.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.615+0000 7fdb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.620+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664492225 0 0) 0x7fdb78137c60 con 0x7fdb781430c0 2024-03-25T18:15:03.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.620+0000 7fdb66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.620+0000 7fdb66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.620+0000 7fdb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.620+0000 7fdb66ffd640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:15:03.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.620+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb58004eb0 con 0x7fdb781430c0 2024-03-25T18:15:03.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.623+0000 7fdb66ffd640 10 client.?.objecter ms_handle_connect 0x7fdb7813f710 2024-03-25T18:15:03.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.623+0000 7fdb66ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:03.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.628+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981788970 0 0) 0x7fdb78139160 con 0x7fdb7813f710 2024-03-25T18:15:03.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.628+0000 7fdb66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.628+0000 7fdb66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.628+0000 7fdb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.628+0000 7fdb66ffd640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:15:03.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.628+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78137c60 con 0x7fdb7813f710 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.632+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820452279 0 0) 0x7fdb781380f0 con 0x7fdb780a98f0 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.632+0000 7fdb66ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.632+0000 7fdb66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.632+0000 7fdb66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.632+0000 7fdb66ffd640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.632+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb78139160 con 0x7fdb780a98f0 2024-03-25T18:15:03.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1583500771 0 0) 0x7fdb58004eb0 con 0x7fdb781430c0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb781380f0 con 0x7fdb781430c0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdb70001ec0 con 0x7fdb781430c0 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:03.634 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.633+0000 7fdb66ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.643+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563452846 0 0) 0x7fdb78137c60 con 0x7fdb7813f710 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.643+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb58004eb0 con 0x7fdb7813f710 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.643+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdb6c001970 con 0x7fdb7813f710 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.643+0000 7fdb66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.643+0000 7fdb66ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.643+0000 7fdb66ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.644 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.645 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.645 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:03.645 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:03.645 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:03.645 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:03.645 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1457249109 0 0) 0x7fdb781380f0 con 0x7fdb781430c0 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 >> v1:172.21.15.132:6790/0 conn(0x7fdb780a98f0 legacy=0x7fdb781dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 >> v1:172.21.15.161:6789/0 conn(0x7fdb7813f710 legacy=0x7fdb78133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 1 monclient: found mon.a 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb781397f0 con 0x7fdb781430c0 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:03.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 10 monclient: _renew_subs 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb781db280 con 0x7fdb781430c0 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 10 monclient: _renew_subs 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb78133960 con 0x7fdb781430c0 2024-03-25T18:15:03.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.654+0000 7fdb7c87b640 1 librados: init done 2024-03-25T18:15:03.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.655+0000 7fdb7c87b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.655+0000 7fdb7c87b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.655+0000 7fdb7c87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:03.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.655+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb78139a40 con 0x7fdb781430c0 2024-03-25T18:15:03.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb70004e10 con 0x7fdb781430c0 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdb70002100 con 0x7fdb781430c0 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb66ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb66ffd640 10 monclient: dump: 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:03.964 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:03.965 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:03.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.963+0000 7fdb74d99640 4 set_mon_vals no callback set 2024-03-25T18:15:03.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.964+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fdb7001af90 con 0x7fdb781430c0 2024-03-25T18:15:03.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.964+0000 7fdb7659c640 10 monclient: get_auth_request con 0x7fdb5806df70 auth_method 0 2024-03-25T18:15:03.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.965+0000 7fdb66ffd640 10 client.4290.objecter ms_handle_connect 0x7fdb5806df70 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fdb70054b90 con 0x7fdb781430c0 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 10 client.4290.objecter ms_dispatch 0x7fdb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb700504d0 con 0x7fdb781430c0 2024-03-25T18:15:03.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:03.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:03.971+0000 7fdb66ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:04.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.224+0000 7fdb7c87b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.224+0000 7fdb7c87b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.224+0000 7fdb7c87b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:04.225 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.224+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb780627b0 con 0x7fdb781430c0 2024-03-25T18:15:04.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.302+0000 7fdb66ffd640 1 -- 172.21.15.132:0/2583004187 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fdb700525d0 con 0x7fdb781430c0 2024-03-25T18:15:04.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.302+0000 7fdb66ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.302+0000 7fdb66ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:04.304 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:04.304 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:04.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 10 librados: watch_flush enter 2024-03-25T18:15:04.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 10 librados: watch_flush exit 2024-03-25T18:15:04.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 10 client.4290.objecter successfully canceled tick 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 >> v1:172.21.15.161:6800/411950978 conn(0x7fdb5806df70 legacy=0x7fdb58070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 10 monclient: shutdown 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 >> v1:172.21.15.132:6789/0 conn(0x7fdb781430c0 legacy=0x7fdb781e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 shutdown_connections 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 >> 172.21.15.132:0/2583004187 conn(0x7fdb780b7ea0 msgr2=0x7fdb7813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 shutdown_connections 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 -- 172.21.15.132:0/2583004187 wait complete. 2024-03-25T18:15:04.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:04.305+0000 7fdb7c87b640 1 librados: shutdown 2024-03-25T18:15:04.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:09.220 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:54.484 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.227 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:54.492 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.280 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:01:54.544 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.280 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:54.522 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.288 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:54.530 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.289 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:54.553 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.293 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:54.558 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.322 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:01:54.564 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:09.324 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:01:54.588 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:11.125 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.236+0000 7f0bd6057640 10 monclient: get_monmap_and_config 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.236+0000 7f0bd6057640 10 monclient: build_initial_monmap 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.236+0000 7f0bd6057640 10 monclient: monmap: 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:11.237524+0000 2024-03-25T18:15:11.237 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:11.237524+0000 2024-03-25T18:15:11.238 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:11.238 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:11.238 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:11.238 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:11.238 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:11.238 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.237+0000 7f0bd6057640 1 Processor -- start 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.237+0000 7f0bd6057640 1 -- start start 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.237+0000 7f0bd6057640 10 monclient: init 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.237+0000 7f0bd6057640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.237+0000 7f0bd6057640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd00a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd01430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:11.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd0142520 con 0x7f0bd013f710 2024-03-25T18:15:11.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00b28a0 con 0x7f0bd01430c0 2024-03-25T18:15:11.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00b2a30 con 0x7f0bd00a98f0 2024-03-25T18:15:11.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:11.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.238+0000 7f0bd6057640 10 monclient(hunting): authenticate will time out at 7351.500818s 2024-03-25T18:15:11.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.243+0000 7f0bcf7fe640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f0bd00a98f0 0x7f0bd00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:36564/0 (socket says 172.21.15.132:36564) 2024-03-25T18:15:11.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.243+0000 7f0bcf7fe640 1 -- 172.21.15.132:0/870476828 learned_addr learned my addr 172.21.15.132:0/870476828 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:11.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.263+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590408123 0 0) 0x7f0bd00b2a30 con 0x7f0bd00a98f0 2024-03-25T18:15:11.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.263+0000 7f0bce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.263+0000 7f0bce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.263+0000 7f0bce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.263+0000 7f0bce7fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:15:11.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.263+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8003520 con 0x7f0bd00a98f0 2024-03-25T18:15:11.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.265+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44951729 0 0) 0x7f0bd0142520 con 0x7f0bd013f710 2024-03-25T18:15:11.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.265+0000 7f0bce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.265+0000 7f0bce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.265+0000 7f0bce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.265+0000 7f0bce7fc640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:15:11.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.265+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd00b2a30 con 0x7f0bd013f710 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860019287 0 0) 0x7f0bb8003520 con 0x7f0bd00a98f0 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 >> v1:172.21.15.161:6789/0 conn(0x7f0bd01430c0 legacy=0x7f0bd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 >> v1:172.21.15.132:6789/0 conn(0x7f0bd013f710 legacy=0x7f0bd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 1 monclient: found mon.noname-b 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd00b44c0 con 0x7f0bd00a98f0 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:11.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0bc0001690 con 0x7f0bd00a98f0 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bd6057640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bd6057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bce7fc640 10 monclient: dump: 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:11.359 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.356+0000 7f0bd6057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bc00017f0 con 0x7f0bd00a98f0 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bce7fc640 1 -- 172.21.15.132:0/870476828 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0bc0001c00 con 0x7f0bd00a98f0 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bd6057640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bd6057640 4 set_mon_vals no callback set 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.374+0000 7f0bce7fc640 10 monclient: dump: 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:11.375 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:11.376 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 10 monclient: shutdown 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 1 -- 172.21.15.132:0/870476828 >> v1:172.21.15.132:6790/0 conn(0x7f0bd00a98f0 legacy=0x7f0bd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 1 -- 172.21.15.132:0/870476828 shutdown_connections 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 1 -- 172.21.15.132:0/870476828 >> 172.21.15.132:0/870476828 conn(0x7f0bd00b7ea0 msgr2=0x7f0bd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 1 -- 172.21.15.132:0/870476828 shutdown_connections 2024-03-25T18:15:11.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.375+0000 7f0bd6057640 1 -- 172.21.15.132:0/870476828 wait complete. 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 10 monclient: build_initial_monmap 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 10 monclient: monmap: 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 1 librados: starting msgr at 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 1 librados: starting objecter 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 1 Processor -- start 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 1 -- start start 2024-03-25T18:15:11.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 1 librados: setting wanted keys 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 1 librados: calling monclient init 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.376+0000 7f0bd6057640 10 monclient: init 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd00a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd01430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00abd30 con 0x7f0bd013f710 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd00abfa0 con 0x7f0bd00a98f0 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd01d77a0 con 0x7f0bd01430c0 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:11.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.377+0000 7f0bd6057640 10 monclient(hunting): authenticate will time out at 7351.639980s 2024-03-25T18:15:11.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.395+0000 7f0bcffff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f0bd01430c0 0x7f0bd01385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:36614/0 (socket says 172.21.15.132:36614) 2024-03-25T18:15:11.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.395+0000 7f0bcffff640 1 -- 172.21.15.132:0/1134295069 learned_addr learned my addr 172.21.15.132:0/1134295069 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:11.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.397+0000 7f0bb7fff640 10 client.?.objecter ms_handle_connect 0x7f0bd01430c0 2024-03-25T18:15:11.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.397+0000 7f0bb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:11.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.420+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633859700 0 0) 0x7f0bd01d77a0 con 0x7f0bd01430c0 2024-03-25T18:15:11.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.420+0000 7f0bb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.420+0000 7f0bb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.420+0000 7f0bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.420+0000 7f0bb7fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:15:11.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.420+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc004eb0 con 0x7f0bd01430c0 2024-03-25T18:15:11.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.428+0000 7f0bb7fff640 10 client.?.objecter ms_handle_connect 0x7f0bd013f710 2024-03-25T18:15:11.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.428+0000 7f0bb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:11.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.431+0000 7f0bb7fff640 10 client.?.objecter ms_handle_connect 0x7f0bd00a98f0 2024-03-25T18:15:11.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.431+0000 7f0bb7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 841259500 0 0) 0x7f0bbc004eb0 con 0x7f0bd01430c0 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bd01d77a0 con 0x7f0bd01430c0 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0bc8001c90 con 0x7f0bd01430c0 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 10 monclient(hunting): dump: 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:11.439 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.438+0000 7f0bb7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:11.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.444+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189712836 0 0) 0x7f0bd00abfa0 con 0x7f0bd00a98f0 2024-03-25T18:15:11.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.444+0000 7f0bb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.444+0000 7f0bb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.444+0000 7f0bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.444+0000 7f0bb7fff640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:15:11.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.444+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc004eb0 con 0x7f0bd00a98f0 2024-03-25T18:15:11.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.455+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795250587 0 0) 0x7f0bd00abd30 con 0x7f0bd013f710 2024-03-25T18:15:11.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.455+0000 7f0bb7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.455+0000 7f0bb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.455+0000 7f0bb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.455+0000 7f0bb7fff640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:15:11.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.455+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bd00abfa0 con 0x7f0bd013f710 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.457+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1347595736 0 0) 0x7f0bbc004eb0 con 0x7f0bd00a98f0 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.457+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bd00abd30 con 0x7f0bd00a98f0 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.457+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0bc0001b00 con 0x7f0bd00a98f0 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.457+0000 7f0bb7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.457+0000 7f0bb7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.457+0000 7f0bb7fff640 10 monclient(hunting): dump: 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:11.458 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:11.459 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:11.459 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:11.459 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2765634785 0 0) 0x7f0bd01d77a0 con 0x7f0bd01430c0 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 >> v1:172.21.15.161:6789/0 conn(0x7f0bd00a98f0 legacy=0x7f0bd01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 >> v1:172.21.15.132:6789/0 conn(0x7f0bd013f710 legacy=0x7f0bd01d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 1 monclient: found mon.c 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd01d83d0 con 0x7f0bd01430c0 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.459+0000 7f0bb7fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bb7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 10 monclient: _renew_subs 2024-03-25T18:15:11.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd01d41e0 con 0x7f0bd01430c0 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 10 monclient: _renew_subs 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd01d5110 con 0x7f0bd01430c0 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.460+0000 7f0bd6057640 1 librados: init done 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.461+0000 7f0bd6057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.461+0000 7f0bd6057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.461+0000 7f0bd6057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:11.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.461+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd01d7930 con 0x7f0bd01430c0 2024-03-25T18:15:11.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bc8003990 con 0x7f0bd01430c0 2024-03-25T18:15:11.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:11.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0bc8003d20 con 0x7f0bd01430c0 2024-03-25T18:15:11.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:11.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bb7fff640 10 monclient: dump: 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:11.793 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.791+0000 7f0bcdffb640 4 set_mon_vals no callback set 2024-03-25T18:15:11.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.792+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f0bc8003f40 con 0x7f0bd01430c0 2024-03-25T18:15:11.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.793+0000 7f0bcf7fe640 10 monclient: get_auth_request con 0x7f0bbc06e040 auth_method 0 2024-03-25T18:15:11.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.793+0000 7f0bb7fff640 10 client.4289.objecter ms_handle_connect 0x7f0bbc06e040 2024-03-25T18:15:11.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f0bc8054ae0 con 0x7f0bd01430c0 2024-03-25T18:15:11.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 10 client.4289.objecter ms_dispatch 0x7f0bd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:11.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:11.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:11.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-25T18:15:11.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bc80505f0 con 0x7f0bd01430c0 2024-03-25T18:15:11.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:11.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:11.798+0000 7f0bb7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:12.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.058+0000 7f0bd6057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:12.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.058+0000 7f0bd6057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:12.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.058+0000 7f0bd6057640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:12.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.058+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bd0136a20 con 0x7f0bd01430c0 2024-03-25T18:15:12.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.090+0000 7f0bb7fff640 1 -- 172.21.15.132:0/1134295069 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f0bc8052520 con 0x7f0bd01430c0 2024-03-25T18:15:12.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.091+0000 7f0bb7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:12.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.091+0000 7f0bb7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:12.092 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:12.092 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:12.094 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 10 librados: watch_flush enter 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 10 librados: watch_flush exit 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 10 client.4289.objecter successfully canceled tick 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 >> v1:172.21.15.161:6800/411950978 conn(0x7f0bbc06e040 legacy=0x7f0bbc070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 10 monclient: shutdown 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 >> v1:172.21.15.132:6790/0 conn(0x7f0bd01430c0 legacy=0x7f0bd01385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 shutdown_connections 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 >> 172.21.15.132:0/1134295069 conn(0x7f0bd00b7ea0 msgr2=0x7f0bd013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 shutdown_connections 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 -- 172.21.15.132:0/1134295069 wait complete. 2024-03-25T18:15:12.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:12.093+0000 7f0bd6057640 1 librados: shutdown 2024-03-25T18:15:12.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:18.913 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:19.026 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.025+0000 7ffb520b3640 10 monclient: get_monmap_and_config 2024-03-25T18:15:19.026 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.025+0000 7ffb520b3640 10 monclient: build_initial_monmap 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.026+0000 7ffb520b3640 10 monclient: monmap: 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:19.027088+0000 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:19.027088+0000 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:19.027 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.026+0000 7ffb520b3640 1 Processor -- start 2024-03-25T18:15:19.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.026+0000 7ffb520b3640 1 -- start start 2024-03-25T18:15:19.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.026+0000 7ffb520b3640 10 monclient: init 2024-03-25T18:15:19.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb4c0a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb4c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c142520 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0b28a0 con 0x7ffb4c13f710 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0b2a30 con 0x7ffb4c1430c0 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:19.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.027+0000 7ffb520b3640 10 monclient(hunting): authenticate will time out at 7359.290389s 2024-03-25T18:15:19.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.039+0000 7ffb4b7fe640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7ffb4c0a98f0 0x7ffb4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:59348/0 (socket says 172.21.15.132:59348) 2024-03-25T18:15:19.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.039+0000 7ffb4b7fe640 1 -- 172.21.15.132:0/1527757667 learned_addr learned my addr 172.21.15.132:0/1527757667 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:19.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.053+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630238449 0 0) 0x7ffb4c142520 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.053+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.053+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.053+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.053+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:15:19.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.053+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb34003520 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253923374 0 0) 0x7ffb4c0b28a0 con 0x7ffb4c13f710 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c142520 con 0x7ffb4c13f710 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952477490 0 0) 0x7ffb4c0b2a30 con 0x7ffb4c1430c0 2024-03-25T18:15:19.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:15:19.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.056+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c0b28a0 con 0x7ffb4c1430c0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2180670931 0 0) 0x7ffb34003520 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 >> v1:172.21.15.132:6790/0 conn(0x7ffb4c1430c0 legacy=0x7ffb4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 >> v1:172.21.15.161:6789/0 conn(0x7ffb4c13f710 legacy=0x7ffb4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0b44c0 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7ffb3c001690 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb4a7fc640 10 monclient: dump: 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb520b3640 5 monclient: authenticate success, global_id 4299 2024-03-25T18:15:19.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.060+0000 7ffb520b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb3c0017f0 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb4a7fc640 1 -- 172.21.15.132:0/1527757667 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7ffb3c001c00 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb4a7fc640 10 monclient: dump: 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.066 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:19.067 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:19.067 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:19.067 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:19.067 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb520b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:19.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.065+0000 7ffb520b3640 4 set_mon_vals no callback set 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 10 monclient: shutdown 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 1 -- 172.21.15.132:0/1527757667 >> v1:172.21.15.132:6789/0 conn(0x7ffb4c0a98f0 legacy=0x7ffb4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 1 -- 172.21.15.132:0/1527757667 shutdown_connections 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 1 -- 172.21.15.132:0/1527757667 >> 172.21.15.132:0/1527757667 conn(0x7ffb4c0b7ea0 msgr2=0x7ffb4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 1 -- 172.21.15.132:0/1527757667 shutdown_connections 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.066+0000 7ffb520b3640 1 -- 172.21.15.132:0/1527757667 wait complete. 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 10 monclient: build_initial_monmap 2024-03-25T18:15:19.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 10 monclient: monmap: 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 1 librados: starting msgr at 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 1 librados: starting objecter 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 1 Processor -- start 2024-03-25T18:15:19.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 1 -- start start 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 1 librados: setting wanted keys 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 1 librados: calling monclient init 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.067+0000 7ffb520b3640 10 monclient: init 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb4c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb4c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb4c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c1db2f0 con 0x7ffb4c13f710 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c139180 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c137670 con 0x7ffb4c1430c0 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:19.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.068+0000 7ffb520b3640 10 monclient(hunting): authenticate will time out at 7359.331000s 2024-03-25T18:15:19.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.078+0000 7ffb4affd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7ffb4c13f710 0x7ffb4c132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:59364/0 (socket says 172.21.15.132:59364) 2024-03-25T18:15:19.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.078+0000 7ffb4affd640 1 -- 172.21.15.132:0/1403196011 learned_addr learned my addr 172.21.15.132:0/1403196011 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:19.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.079+0000 7ffb33fff640 10 client.?.objecter ms_handle_connect 0x7ffb4c13f710 2024-03-25T18:15:19.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.079+0000 7ffb33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:19.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.087+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798126572 0 0) 0x7ffb4c1db2f0 con 0x7ffb4c13f710 2024-03-25T18:15:19.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.087+0000 7ffb33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.087+0000 7ffb33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.087+0000 7ffb33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.087+0000 7ffb33fff640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:15:19.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.087+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb38004eb0 con 0x7ffb4c13f710 2024-03-25T18:15:19.090 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.089+0000 7ffb33fff640 10 client.?.objecter ms_handle_connect 0x7ffb4c1430c0 2024-03-25T18:15:19.090 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.089+0000 7ffb33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:19.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.090+0000 7ffb33fff640 10 client.?.objecter ms_handle_connect 0x7ffb4c0a98f0 2024-03-25T18:15:19.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.090+0000 7ffb33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:19.096 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.096+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608511326 0 0) 0x7ffb4c139180 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.096+0000 7ffb33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.096+0000 7ffb33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.096+0000 7ffb33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.096+0000 7ffb33fff640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:15:19.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.096+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c1db2f0 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.105+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205760435 0 0) 0x7ffb4c137670 con 0x7ffb4c1430c0 2024-03-25T18:15:19.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.105+0000 7ffb33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.105+0000 7ffb33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.105+0000 7ffb33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.105+0000 7ffb33fff640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:15:19.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.105+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c139180 con 0x7ffb4c1430c0 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 625273463 0 0) 0x7ffb38004eb0 con 0x7ffb4c13f710 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb4c137670 con 0x7ffb4c13f710 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7ffb40001a30 con 0x7ffb4c13f710 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 10 monclient(hunting): dump: 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.111 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.112 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.110+0000 7ffb33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.114+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598939457 0 0) 0x7ffb4c1db2f0 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.114+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb38004eb0 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.114+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7ffb3c001b00 con 0x7ffb4c0a98f0 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.114+0000 7ffb33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.114+0000 7ffb33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.114+0000 7ffb33fff640 10 monclient(hunting): dump: 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:19.115 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:19.116 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:19.116 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:19.116 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 113517225 0 0) 0x7ffb4c137670 con 0x7ffb4c13f710 2024-03-25T18:15:19.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 >> v1:172.21.15.132:6790/0 conn(0x7ffb4c1430c0 legacy=0x7ffb4c1e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 >> v1:172.21.15.161:6789/0 conn(0x7ffb4c0a98f0 legacy=0x7ffb4c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 1 monclient: found mon.a 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c137200 con 0x7ffb4c13f710 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 10 monclient: _renew_subs 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c0b7290 con 0x7ffb4c13f710 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 10 monclient: _renew_subs 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:19.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c133680 con 0x7ffb4c13f710 2024-03-25T18:15:19.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.121+0000 7ffb520b3640 1 librados: init done 2024-03-25T18:15:19.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.122+0000 7ffb520b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:19.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.122+0000 7ffb520b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:19.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.122+0000 7ffb520b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:19.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.122+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c137800 con 0x7ffb4c13f710 2024-03-25T18:15:19.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.129+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb40001c70 con 0x7ffb4c13f710 2024-03-25T18:15:19.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.129+0000 7ffb33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:19.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.129+0000 7ffb49ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:19.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.424+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7ffb40003a60 con 0x7ffb4c13f710 2024-03-25T18:15:19.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.424+0000 7ffb33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.424+0000 7ffb33fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:19.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.424+0000 7ffb33fff640 10 monclient: dump: 2024-03-25T18:15:19.425 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:19.426 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:19.427 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.425+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7ffb40003c40 con 0x7ffb4c13f710 2024-03-25T18:15:19.427 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.426+0000 7ffb4b7fe640 10 monclient: get_auth_request con 0x7ffb3806df70 auth_method 0 2024-03-25T18:15:19.427 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.426+0000 7ffb33fff640 10 client.4302.objecter ms_handle_connect 0x7ffb3806df70 2024-03-25T18:15:19.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7ffb400548e0 con 0x7ffb4c13f710 2024-03-25T18:15:19.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 10 client.4302.objecter ms_dispatch 0x7ffb4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:19.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 3 client.4302.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:19.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 3 client.4302.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:19.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 20 client.4302.objecter dump_active .. 0 homeless 2024-03-25T18:15:19.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb400503f0 con 0x7ffb4c13f710 2024-03-25T18:15:19.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:19.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.429+0000 7ffb33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:19.649 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.648+0000 7ffb520b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.649 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.648+0000 7ffb520b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.649 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.648+0000 7ffb520b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:19.649 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.648+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb4c0627b0 con 0x7ffb4c13f710 2024-03-25T18:15:19.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.711+0000 7ffb33fff640 1 -- 172.21.15.132:0/1403196011 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7ffb40052320 con 0x7ffb4c13f710 2024-03-25T18:15:19.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.711+0000 7ffb33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:19.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.711+0000 7ffb33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:19.712 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:19.712 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:19.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 10 librados: watch_flush enter 2024-03-25T18:15:19.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 10 librados: watch_flush exit 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 20 client.4302.objecter shutdown clearing up homeless session... 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 10 client.4302.objecter successfully canceled tick 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 >> v1:172.21.15.161:6800/411950978 conn(0x7ffb3806df70 legacy=0x7ffb38070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 10 monclient: shutdown 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 >> v1:172.21.15.132:6789/0 conn(0x7ffb4c13f710 legacy=0x7ffb4c132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 shutdown_connections 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 >> 172.21.15.132:0/1403196011 conn(0x7ffb4c0b7ea0 msgr2=0x7ffb4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 shutdown_connections 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 -- 172.21.15.132:0/1403196011 wait complete. 2024-03-25T18:15:19.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:19.714+0000 7ffb520b3640 1 librados: shutdown 2024-03-25T18:15:19.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:26.534 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:26.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.651+0000 7f46d9fb5640 10 monclient: get_monmap_and_config 2024-03-25T18:15:26.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.651+0000 7f46d9fb5640 10 monclient: build_initial_monmap 2024-03-25T18:15:26.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.651+0000 7f46d9fb5640 10 monclient: monmap: 2024-03-25T18:15:26.652 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:26.652 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:26.652 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:26.652745+0000 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:26.652745+0000 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:26.653 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:26.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.652+0000 7f46d9fb5640 1 Processor -- start 2024-03-25T18:15:26.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.652+0000 7f46d9fb5640 1 -- start start 2024-03-25T18:15:26.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.652+0000 7f46d9fb5640 10 monclient: init 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f46d40b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f46d40b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f46d414be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40af940 con 0x7f46d414be90 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40b0a70 con 0x7f46d40b1730 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d40b0d50 con 0x7f46d40b4d60 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:26.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.653+0000 7f46d9fb5640 10 monclient(hunting): authenticate will time out at 7366.916087s 2024-03-25T18:15:26.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.658+0000 7f46d2ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46d40b1730 0x7f46d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:40552/0 (socket says 172.21.15.132:40552) 2024-03-25T18:15:26.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.658+0000 7f46d2ffd640 1 -- 172.21.15.132:0/1168235329 learned_addr learned my addr 172.21.15.132:0/1168235329 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:26.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.673+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763363367 0 0) 0x7f46d40af940 con 0x7f46d414be90 2024-03-25T18:15:26.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.673+0000 7f46d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.673+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.673+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.673+0000 7f46d27fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.673+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46bc003520 con 0x7f46d414be90 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.674+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668200336 0 0) 0x7f46d40b0a70 con 0x7f46d40b1730 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.674+0000 7f46d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.674+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.674+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.674+0000 7f46d27fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:15:26.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.674+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d40af940 con 0x7f46d40b1730 2024-03-25T18:15:26.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.678+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911953440 0 0) 0x7f46d40b0d50 con 0x7f46d40b4d60 2024-03-25T18:15:26.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.678+0000 7f46d27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.678+0000 7f46d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.680 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.678+0000 7f46d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.680 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.678+0000 7f46d27fc640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:15:26.680 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.678+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d40b0a70 con 0x7f46d40b4d60 2024-03-25T18:15:26.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223863716 0 0) 0x7f46bc003520 con 0x7f46d414be90 2024-03-25T18:15:26.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 >> v1:172.21.15.132:6790/0 conn(0x7f46d40b4d60 legacy=0x7f46d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 >> v1:172.21.15.161:6789/0 conn(0x7f46d40b1730 legacy=0x7f46d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:26.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 1 monclient: found mon.noname-a 2024-03-25T18:15:26.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d40b1030 con 0x7f46d414be90 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f46cc001a10 con 0x7f46d414be90 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d27fc640 10 monclient: dump: 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.692 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:26.693 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:26.693 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:26.693 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:26.693 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:26.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d9fb5640 5 monclient: authenticate success, global_id 4305 2024-03-25T18:15:26.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.690+0000 7f46d9fb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46cc001b70 con 0x7f46d414be90 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d27fc640 1 -- 172.21.15.132:0/1168235329 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f46cc001f80 con 0x7f46d414be90 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d27fc640 10 monclient: dump: 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:26.721 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:26.722 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d9fb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:26.722 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d9fb5640 4 set_mon_vals no callback set 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d9fb5640 10 monclient: shutdown 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d9fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.720+0000 7f46d9fb5640 1 -- 172.21.15.132:0/1168235329 >> v1:172.21.15.132:6789/0 conn(0x7f46d414be90 legacy=0x7f46d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.721+0000 7f46d9fb5640 1 -- 172.21.15.132:0/1168235329 shutdown_connections 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.721+0000 7f46d9fb5640 1 -- 172.21.15.132:0/1168235329 >> 172.21.15.132:0/1168235329 conn(0x7f46d40617a0 msgr2=0x7f46d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.721+0000 7f46d9fb5640 1 -- 172.21.15.132:0/1168235329 shutdown_connections 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.721+0000 7f46d9fb5640 1 -- 172.21.15.132:0/1168235329 wait complete. 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.721+0000 7f46d9fb5640 10 monclient: build_initial_monmap 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.721+0000 7f46d9fb5640 10 monclient: monmap: 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:26.724 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 1 librados: starting msgr at 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 1 librados: starting objecter 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 1 Processor -- start 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 1 -- start start 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 1 librados: setting wanted keys 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 1 librados: calling monclient init 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.722+0000 7f46d9fb5640 10 monclient: init 2024-03-25T18:15:26.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient(hunting): picked mon.noname-c con 0x7f46d40b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient(hunting): picked mon.noname-a con 0x7f46d40b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient(hunting): picked mon.noname-b con 0x7f46d414be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4141950 con 0x7f46d40b4d60 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d41d7940 con 0x7f46d414be90 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d41d8570 con 0x7f46d40b1730 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:26.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.723+0000 7f46d9fb5640 10 monclient(hunting): authenticate will time out at 7366.985842s 2024-03-25T18:15:26.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.732+0000 7f46d3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46d414be90 0x7f46d41e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:40606/0 (socket says 172.21.15.132:40606) 2024-03-25T18:15:26.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.732+0000 7f46d3fff640 1 -- 172.21.15.132:0/3559357520 learned_addr learned my addr 172.21.15.132:0/3559357520 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:26.734 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.734+0000 7f46bbfff640 10 client.?.objecter ms_handle_connect 0x7f46d414be90 2024-03-25T18:15:26.735 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.734+0000 7f46bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.738+0000 7f46bbfff640 10 client.?.objecter ms_handle_connect 0x7f46d40b1730 2024-03-25T18:15:26.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.738+0000 7f46bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.742+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123049749 0 0) 0x7f46d41d7940 con 0x7f46d414be90 2024-03-25T18:15:26.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.742+0000 7f46bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.742+0000 7f46bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.742+0000 7f46bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.742+0000 7f46bbfff640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:15:26.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.742+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0004eb0 con 0x7f46d414be90 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913533994 0 0) 0x7f46d41d8570 con 0x7f46d40b1730 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d41d7940 con 0x7f46d40b1730 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 10 client.?.objecter ms_handle_connect 0x7f46d40b4d60 2024-03-25T18:15:26.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.743+0000 7f46bbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:26.753 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.753+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141155982 0 0) 0x7f46d4141950 con 0x7f46d40b4d60 2024-03-25T18:15:26.753 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.753+0000 7f46bbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:26.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.753+0000 7f46bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:26.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.753+0000 7f46bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:26.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.753+0000 7f46bbfff640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:15:26.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.753+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46d41d8570 con 0x7f46d40b4d60 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1772510710 0 0) 0x7f46d41d7940 con 0x7f46d40b1730 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46d4141950 con 0x7f46d40b1730 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f46c40018b0 con 0x7f46d40b1730 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 10 monclient(hunting): dump: 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:26.756 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:26.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.755+0000 7f46bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.766+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4236059321 0 0) 0x7f46c0004eb0 con 0x7f46d414be90 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.766+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46d41d7940 con 0x7f46d414be90 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.766+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f46cc001f40 con 0x7f46d414be90 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.766+0000 7f46bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.766+0000 7f46bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.766+0000 7f46bbfff640 10 monclient(hunting): dump: 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.767 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:26.768 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:26.768 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:26.768 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:26.768 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:26.768 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:26.768 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2923883575 0 0) 0x7f46d4141950 con 0x7f46d40b1730 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 >> v1:172.21.15.161:6789/0 conn(0x7f46d414be90 legacy=0x7f46d41e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 >> v1:172.21.15.132:6789/0 conn(0x7f46d40b4d60 legacy=0x7f46d4142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 1 monclient: found mon.c 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:26.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4138840 con 0x7f46d40b1730 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 10 monclient: _finish_auth 0 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 10 monclient: _renew_subs 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d4142f70 con 0x7f46d40b1730 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 10 monclient: _renew_subs 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d41432c0 con 0x7f46d40b1730 2024-03-25T18:15:26.774 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.772+0000 7f46d9fb5640 1 librados: init done 2024-03-25T18:15:26.775 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.773+0000 7f46d9fb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.775 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.773+0000 7f46d9fb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:26.775 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.773+0000 7f46d9fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:26.775 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:26.773+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d41d7ad0 con 0x7f46d40b1730 2024-03-25T18:15:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46c4003610 con 0x7f46d40b1730 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f46c40039a0 con 0x7f46d40b1730 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46bbfff640 10 monclient: dump: 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:27.074 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:27.075 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:27.075 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:27.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.072+0000 7f46d1ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:27.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.073+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f46c4003bc0 con 0x7f46d40b1730 2024-03-25T18:15:27.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.074+0000 7f46d2ffd640 10 monclient: get_auth_request con 0x7f46c006df70 auth_method 0 2024-03-25T18:15:27.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.074+0000 7f46bbfff640 10 client.4301.objecter ms_handle_connect 0x7f46c006df70 2024-03-25T18:15:27.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f46c4053560 con 0x7f46d40b1730 2024-03-25T18:15:27.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 10 client.4301.objecter ms_dispatch 0x7f46d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:27.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:27.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:27.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T18:15:27.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46c404f070 con 0x7f46d40b1730 2024-03-25T18:15:27.081 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:27.081 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.079+0000 7f46bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:27.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.312+0000 7f46d9fb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.312+0000 7f46d9fb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.312+0000 7f46d9fb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:27.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.312+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46d41389d0 con 0x7f46d40b1730 2024-03-25T18:15:27.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.339+0000 7f46bbfff640 1 -- 172.21.15.132:0/3559357520 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f46c4050fa0 con 0x7f46d40b1730 2024-03-25T18:15:27.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.339+0000 7f46bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.339+0000 7f46bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:27.340 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:27.341 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:27.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 10 librados: watch_flush enter 2024-03-25T18:15:27.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 10 librados: watch_flush exit 2024-03-25T18:15:27.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 10 client.4301.objecter successfully canceled tick 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 >> v1:172.21.15.161:6800/411950978 conn(0x7f46c006df70 legacy=0x7f46c0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 10 monclient: shutdown 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 >> v1:172.21.15.132:6790/0 conn(0x7f46d40b1730 legacy=0x7f46d41d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 shutdown_connections 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 >> 172.21.15.132:0/3559357520 conn(0x7f46d40617a0 msgr2=0x7f46d40b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 shutdown_connections 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 -- 172.21.15.132:0/3559357520 wait complete. 2024-03-25T18:15:27.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:27.342+0000 7f46d9fb5640 1 librados: shutdown 2024-03-25T18:15:27.355 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:34.162 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:34.258 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:19.522 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.262+0000 7fbf24e81640 10 monclient: get_monmap_and_config 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.262+0000 7fbf24e81640 10 monclient: build_initial_monmap 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.262+0000 7fbf24e81640 10 monclient: monmap: 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:34.263518+0000 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:34.263518+0000 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:34.263 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:34.264 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:34.264 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:34.264 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:34.264 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 1 Processor -- start 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 1 -- start start 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient: init 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient(hunting): picked mon.noname-c con 0x7fbf200a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:34.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient(hunting): picked mon.noname-a con 0x7fbf201430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf20142520 con 0x7fbf201430c0 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf200b28a0 con 0x7fbf200a98f0 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf200b2a30 con 0x7fbf2013f710 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.263+0000 7fbf24e81640 10 monclient(hunting): authenticate will time out at 7374.526566s 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.264+0000 7fbf1ed9d640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fbf201430c0 0x7fbf2014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37184/0 (socket says 172.21.15.132:37184) 2024-03-25T18:15:34.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.264+0000 7fbf1ed9d640 1 -- 172.21.15.132:0/838185722 learned_addr learned my addr 172.21.15.132:0/838185722 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:34.273 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:19.538 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.309 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:19.573 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.313 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:19.555 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.317 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:19.558 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.337 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:19.601 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.338+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365964756 0 0) 0x7fbf20142520 con 0x7fbf201430c0 2024-03-25T18:15:34.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.338+0000 7fbf1dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.338+0000 7fbf1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.338+0000 7fbf1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.338+0000 7fbf1dd9b640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:15:34.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.338+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c003520 con 0x7fbf201430c0 2024-03-25T18:15:34.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.346+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907355628 0 0) 0x7fbf200b2a30 con 0x7fbf2013f710 2024-03-25T18:15:34.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.346+0000 7fbf1dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.346+0000 7fbf1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.346+0000 7fbf1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.346+0000 7fbf1dd9b640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:15:34.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.346+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf20142520 con 0x7fbf2013f710 2024-03-25T18:15:34.347 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:19.589 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.349 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:19.614 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.350+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068288079 0 0) 0x7fbf200b28a0 con 0x7fbf200a98f0 2024-03-25T18:15:34.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.350+0000 7fbf1dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.350+0000 7fbf1dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.350+0000 7fbf1dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.350+0000 7fbf1dd9b640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:15:34.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.350+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf200b2a30 con 0x7fbf200a98f0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1158193809 0 0) 0x7fbf0c003520 con 0x7fbf201430c0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 >> v1:172.21.15.132:6790/0 conn(0x7fbf2013f710 legacy=0x7fbf201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 >> v1:172.21.15.161:6789/0 conn(0x7fbf200a98f0 legacy=0x7fbf200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf200b44c0 con 0x7fbf201430c0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbf18001a10 con 0x7fbf201430c0 2024-03-25T18:15:34.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf1dd9b640 10 monclient: dump: 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:34.357 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:34.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf24e81640 5 monclient: authenticate success, global_id 4311 2024-03-25T18:15:34.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.354+0000 7fbf24e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:34.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf18001b70 con 0x7fbf201430c0 2024-03-25T18:15:34.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf1dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:34.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf1dd9b640 1 -- 172.21.15.132:0/838185722 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbf18001f80 con 0x7fbf201430c0 2024-03-25T18:15:34.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf24e81640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:34.366 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf24e81640 4 set_mon_vals no callback set 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf1dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf1dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.365+0000 7fbf1dd9b640 10 monclient: dump: 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:34.367 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:34.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 10 monclient: shutdown 2024-03-25T18:15:34.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:34.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 1 -- 172.21.15.132:0/838185722 >> v1:172.21.15.132:6789/0 conn(0x7fbf201430c0 legacy=0x7fbf2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 1 -- 172.21.15.132:0/838185722 shutdown_connections 2024-03-25T18:15:34.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 1 -- 172.21.15.132:0/838185722 >> 172.21.15.132:0/838185722 conn(0x7fbf200b7ea0 msgr2=0x7fbf200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:34.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 1 -- 172.21.15.132:0/838185722 shutdown_connections 2024-03-25T18:15:34.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.366+0000 7fbf24e81640 1 -- 172.21.15.132:0/838185722 wait complete. 2024-03-25T18:15:34.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.367+0000 7fbf24e81640 10 monclient: build_initial_monmap 2024-03-25T18:15:34.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.367+0000 7fbf24e81640 10 monclient: monmap: 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:34.371 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.367+0000 7fbf24e81640 1 librados: starting msgr at 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.367+0000 7fbf24e81640 1 librados: starting objecter 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.367+0000 7fbf24e81640 1 Processor -- start 2024-03-25T18:15:34.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 1 -- start start 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 1 librados: setting wanted keys 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 1 librados: calling monclient init 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient: init 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient(hunting): picked mon.noname-a con 0x7fbf200a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient(hunting): picked mon.noname-c con 0x7fbf2013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient(hunting): picked mon.noname-b con 0x7fbf201430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:34.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf200abb00 con 0x7fbf200a98f0 2024-03-25T18:15:34.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf200abde0 con 0x7fbf201430c0 2024-03-25T18:15:34.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf20138e20 con 0x7fbf2013f710 2024-03-25T18:15:34.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:34.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.368+0000 7fbf24e81640 10 monclient(hunting): authenticate will time out at 7374.631481s 2024-03-25T18:15:34.386 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:19.651 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.388+0000 7fbf1ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbf201430c0 0x7fbf200ac200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:59492/0 (socket says 172.21.15.132:59492) 2024-03-25T18:15:34.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.388+0000 7fbf1ed9d640 1 -- 172.21.15.132:0/1336358424 learned_addr learned my addr 172.21.15.132:0/1336358424 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:34.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.391+0000 7fbf177fe640 10 client.?.objecter ms_handle_connect 0x7fbf201430c0 2024-03-25T18:15:34.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.391+0000 7fbf177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.394+0000 7fbf177fe640 10 client.?.objecter ms_handle_connect 0x7fbf2013f710 2024-03-25T18:15:34.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.394+0000 7fbf177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.397 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.396+0000 7fbf177fe640 10 client.?.objecter ms_handle_connect 0x7fbf200a98f0 2024-03-25T18:15:34.397 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.396+0000 7fbf177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.415+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139706810 0 0) 0x7fbf20138e20 con 0x7fbf2013f710 2024-03-25T18:15:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.415+0000 7fbf177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.415+0000 7fbf177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.415+0000 7fbf177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.415+0000 7fbf177fe640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:15:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.415+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf00004eb0 con 0x7fbf2013f710 2024-03-25T18:15:34.426 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.425+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3047031312 0 0) 0x7fbf00004eb0 con 0x7fbf2013f710 2024-03-25T18:15:34.426 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.425+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf20138e20 con 0x7fbf2013f710 2024-03-25T18:15:34.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.429+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845308318 0 0) 0x7fbf200abde0 con 0x7fbf201430c0 2024-03-25T18:15:34.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.429+0000 7fbf177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.429+0000 7fbf177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.429+0000 7fbf177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.429+0000 7fbf177fe640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:15:34.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.429+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf00004eb0 con 0x7fbf201430c0 2024-03-25T18:15:34.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.436+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbf08001a70 con 0x7fbf2013f710 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.436+0000 7fbf177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.436+0000 7fbf177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.436+0000 7fbf177fe640 10 monclient(hunting): dump: 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:34.437 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:34.438 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.436+0000 7fbf177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.438+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224444858 0 0) 0x7fbf200abb00 con 0x7fbf200a98f0 2024-03-25T18:15:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.438+0000 7fbf177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.438+0000 7fbf177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.438+0000 7fbf177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.438+0000 7fbf177fe640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:15:34.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.439+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf200abde0 con 0x7fbf200a98f0 2024-03-25T18:15:34.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2494084447 0 0) 0x7fbf20138e20 con 0x7fbf2013f710 2024-03-25T18:15:34.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 >> v1:172.21.15.161:6789/0 conn(0x7fbf201430c0 legacy=0x7fbf200ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 >> v1:172.21.15.132:6789/0 conn(0x7fbf200a98f0 legacy=0x7fbf201385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:34.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 1 monclient: found mon.c 2024-03-25T18:15:34.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf20137b00 con 0x7fbf2013f710 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 5 monclient: authenticate success, global_id 4307 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 10 monclient: _renew_subs 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf201e34b0 con 0x7fbf2013f710 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 10 monclient: _renew_subs 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf201e3820 con 0x7fbf2013f710 2024-03-25T18:15:34.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.445+0000 7fbf24e81640 1 librados: init done 2024-03-25T18:15:34.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.446+0000 7fbf24e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.446+0000 7fbf24e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.446+0000 7fbf24e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:34.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.446+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf200627b0 con 0x7fbf2013f710 2024-03-25T18:15:34.762 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf08001bd0 con 0x7fbf2013f710 2024-03-25T18:15:34.762 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:34.762 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbf08003a60 con 0x7fbf2013f710 2024-03-25T18:15:34.762 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.762 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf177fe640 10 monclient: dump: 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:34.763 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.761+0000 7fbf1d59a640 4 set_mon_vals no callback set 2024-03-25T18:15:34.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.762+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fbf08019a30 con 0x7fbf2013f710 2024-03-25T18:15:34.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.763+0000 7fbf1e59c640 10 monclient: get_auth_request con 0x7fbf0006dd90 auth_method 0 2024-03-25T18:15:34.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.763+0000 7fbf177fe640 10 client.4307.objecter ms_handle_connect 0x7fbf0006dd90 2024-03-25T18:15:34.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.767+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fbf08053690 con 0x7fbf2013f710 2024-03-25T18:15:34.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.767+0000 7fbf177fe640 10 client.4307.objecter ms_dispatch 0x7fbf20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:34.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.767+0000 7fbf177fe640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:34.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.767+0000 7fbf177fe640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:34.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.768+0000 7fbf177fe640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-25T18:15:34.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.768+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf0804efd0 con 0x7fbf2013f710 2024-03-25T18:15:34.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.768+0000 7fbf177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:34.768+0000 7fbf177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:35.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.014+0000 7fbf24e81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:35.016 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.014+0000 7fbf24e81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:35.016 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.014+0000 7fbf24e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:35.016 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.014+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf200b1760 con 0x7fbf2013f710 2024-03-25T18:15:35.053 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.052+0000 7fbf177fe640 1 -- 172.21.15.132:0/1336358424 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fbf080510d0 con 0x7fbf2013f710 2024-03-25T18:15:35.053 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.052+0000 7fbf177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:35.053 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.052+0000 7fbf177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:35.053 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:35.053 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:35.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.054+0000 7fbf24e81640 10 librados: watch_flush enter 2024-03-25T18:15:35.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.054+0000 7fbf24e81640 10 librados: watch_flush exit 2024-03-25T18:15:35.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.054+0000 7fbf24e81640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-25T18:15:35.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.054+0000 7fbf24e81640 10 client.4307.objecter successfully canceled tick 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 >> v1:172.21.15.161:6800/411950978 conn(0x7fbf0006dd90 legacy=0x7fbf00070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 10 monclient: shutdown 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 >> v1:172.21.15.132:6790/0 conn(0x7fbf2013f710 legacy=0x7fbf20136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 shutdown_connections 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 >> 172.21.15.132:0/1336358424 conn(0x7fbf200b7ea0 msgr2=0x7fbf2014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 shutdown_connections 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 -- 172.21.15.132:0/1336358424 wait complete. 2024-03-25T18:15:35.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:35.055+0000 7fbf24e81640 1 librados: shutdown 2024-03-25T18:15:35.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:41.876 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:41.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.995+0000 7fdce1ee4640 10 monclient: get_monmap_and_config 2024-03-25T18:15:41.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.995+0000 7fdce1ee4640 10 monclient: build_initial_monmap 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.995+0000 7fdce1ee4640 10 monclient: monmap: 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:41.996979+0000 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:41.996979+0000 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:41.997 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.996+0000 7fdce1ee4640 1 Processor -- start 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 1 -- start start 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient: init 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient(hunting): picked mon.noname-b con 0x7fdcdc0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:41.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient(hunting): picked mon.noname-a con 0x7fdcdc0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:42.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient(hunting): picked mon.noname-c con 0x7fdcdc14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:42.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0af940 con 0x7fdcdc0b1730 2024-03-25T18:15:42.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0b0a70 con 0x7fdcdc14be90 2024-03-25T18:15:42.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc0b0d50 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:42.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:41.997+0000 7fdce1ee4640 10 monclient(hunting): authenticate will time out at 7382.260405s 2024-03-25T18:15:42.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.003+0000 7fdcdbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdcdc14be90 0x7fdcdc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:35842/0 (socket says 172.21.15.132:35842) 2024-03-25T18:15:42.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.003+0000 7fdcdbfff640 1 -- 172.21.15.132:0/1817557947 learned_addr learned my addr 172.21.15.132:0/1817557947 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:42.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.015+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037139029 0 0) 0x7fdcdc0b0a70 con 0x7fdcdc14be90 2024-03-25T18:15:42.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.015+0000 7fdcdaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.015 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.015+0000 7fdcdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.016 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.015+0000 7fdcdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.016 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.015+0000 7fdcdaffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:15:42.016 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.015+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc8003520 con 0x7fdcdc14be90 2024-03-25T18:15:42.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.016+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703234297 0 0) 0x7fdcdc0b0d50 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.016+0000 7fdcdaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.016+0000 7fdcdaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.016+0000 7fdcdaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.016+0000 7fdcdaffd640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:15:42.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.016+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcdc0b0a70 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519462416 0 0) 0x7fdcdc0b0a70 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 >> v1:172.21.15.161:6789/0 conn(0x7fdcdc14be90 legacy=0x7fdcdc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 >> v1:172.21.15.132:6789/0 conn(0x7fdcdc0b1730 legacy=0x7fdcdc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 1 monclient: found mon.noname-b 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc0b1030 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:42.027 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdccc0015f0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdcdaffd640 10 monclient: dump: 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:42.028 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:42.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdce1ee4640 5 monclient: authenticate success, global_id 4310 2024-03-25T18:15:42.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.026+0000 7fdce1ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdccc001750 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdcdaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdcdaffd640 1 -- 172.21.15.132:0/1817557947 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdccc001ba0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdce1ee4640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdce1ee4640 4 set_mon_vals no callback set 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdcdaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdcdaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:42.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.040+0000 7fdcdaffd640 10 monclient: dump: 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:42.042 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:42.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 10 monclient: shutdown 2024-03-25T18:15:42.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:42.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 1 -- 172.21.15.132:0/1817557947 >> v1:172.21.15.132:6790/0 conn(0x7fdcdc0b4d60 legacy=0x7fdcdc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 1 -- 172.21.15.132:0/1817557947 shutdown_connections 2024-03-25T18:15:42.044 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 1 -- 172.21.15.132:0/1817557947 >> 172.21.15.132:0/1817557947 conn(0x7fdcdc0617a0 msgr2=0x7fdcdc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 1 -- 172.21.15.132:0/1817557947 shutdown_connections 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.041+0000 7fdce1ee4640 1 -- 172.21.15.132:0/1817557947 wait complete. 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 10 monclient: build_initial_monmap 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 10 monclient: monmap: 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 1 librados: starting msgr at 2024-03-25T18:15:42.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 1 librados: starting objecter 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 1 Processor -- start 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 1 -- start start 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 1 librados: setting wanted keys 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 1 librados: calling monclient init 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.042+0000 7fdce1ee4640 10 monclient: init 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient(hunting): picked mon.noname-a con 0x7fdcdc0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient(hunting): picked mon.noname-c con 0x7fdcdc0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient(hunting): picked mon.noname-b con 0x7fdcdc14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc12da80 con 0x7fdcdc0b1730 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc12d790 con 0x7fdcdc14be90 2024-03-25T18:15:42.046 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcdc12f940 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.047 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:42.047 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.043+0000 7fdce1ee4640 10 monclient(hunting): authenticate will time out at 7382.306222s 2024-03-25T18:15:42.052 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.051+0000 7fdcdbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdcdc14be90 0x7fdcdc12d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:35878/0 (socket says 172.21.15.132:35878) 2024-03-25T18:15:42.052 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.051+0000 7fdcdbfff640 1 -- 172.21.15.132:0/3004940090 learned_addr learned my addr 172.21.15.132:0/3004940090 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:42.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.056+0000 7fdcd8ff9640 10 client.?.objecter ms_handle_connect 0x7fdcdc14be90 2024-03-25T18:15:42.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.056+0000 7fdcd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:42.060 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.059+0000 7fdcd8ff9640 10 client.?.objecter ms_handle_connect 0x7fdcdc0b4d60 2024-03-25T18:15:42.060 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.059+0000 7fdcd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:42.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.062+0000 7fdcd8ff9640 10 client.?.objecter ms_handle_connect 0x7fdcdc0b1730 2024-03-25T18:15:42.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.062+0000 7fdcd8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.068+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18799884 0 0) 0x7fdcdc12d790 con 0x7fdcdc14be90 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.068+0000 7fdcd8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.068+0000 7fdcd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.068+0000 7fdcd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.068+0000 7fdcd8ff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:15:42.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.068+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcbc004eb0 con 0x7fdcdc14be90 2024-03-25T18:15:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.071+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1263052663 0 0) 0x7fdcdc12f940 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.071+0000 7fdcd8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.071+0000 7fdcd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.071+0000 7fdcd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.071+0000 7fdcd8ff9640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:15:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.071+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcdc12d790 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.083+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171463946 0 0) 0x7fdcdc12da80 con 0x7fdcdc0b1730 2024-03-25T18:15:42.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.083+0000 7fdcd8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.083+0000 7fdcd8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.083+0000 7fdcd8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.083+0000 7fdcd8ff9640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:15:42.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.083+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcdc12f940 con 0x7fdcdc0b1730 2024-03-25T18:15:42.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1469036662 0 0) 0x7fdcdc12d790 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcdc12da80 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdcc40018a0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:42.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.087+0000 7fdcd8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:42.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.097+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 26922511 0 0) 0x7fdcbc004eb0 con 0x7fdcdc14be90 2024-03-25T18:15:42.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.097+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcdc12d790 con 0x7fdcdc14be90 2024-03-25T18:15:42.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.097+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdcd4001c10 con 0x7fdcdc14be90 2024-03-25T18:15:42.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.097+0000 7fdcd8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.097+0000 7fdcd8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.097+0000 7fdcd8ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:42.099 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:42.100 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 472624166 0 0) 0x7fdcdc12da80 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 >> v1:172.21.15.161:6789/0 conn(0x7fdcdc14be90 legacy=0x7fdcdc12d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 >> v1:172.21.15.132:6789/0 conn(0x7fdcdc0b1730 legacy=0x7fdcdc135e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 1 monclient: found mon.c 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcdc12e1f0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:42.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdcd8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 10 monclient: _renew_subs 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcdc132d50 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 10 monclient: _renew_subs 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcdc12e670 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.098+0000 7fdce1ee4640 1 librados: init done 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.099+0000 7fdce1ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.099+0000 7fdce1ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.099+0000 7fdce1ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.099+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcdc0627b0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcc4001d50 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcd8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fdcc4003790 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcd8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcd8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcd8ff9640 10 monclient: dump: 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:42.411 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.410+0000 7fdcda7fc640 4 set_mon_vals no callback set 2024-03-25T18:15:42.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.411+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fdcc4003970 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.411+0000 7fdcdb7fe640 10 monclient: get_auth_request con 0x7fdcbc06df20 auth_method 0 2024-03-25T18:15:42.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.412+0000 7fdcd8ff9640 10 client.4313.objecter ms_handle_connect 0x7fdcbc06df20 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.416+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fdcc4053590 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.416+0000 7fdcd8ff9640 10 client.4313.objecter ms_dispatch 0x7fdcdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.416+0000 7fdcd8ff9640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.416+0000 7fdcd8ff9640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.416+0000 7fdcd8ff9640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.417+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcc404eed0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.417+0000 7fdcd8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.417+0000 7fdcd8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:42.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.615+0000 7fdce1ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.615+0000 7fdce1ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.615+0000 7fdce1ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:42.617 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.615+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcdc069f40 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.652+0000 7fdcd8ff9640 1 -- 172.21.15.132:0/3004940090 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fdcc4050fd0 con 0x7fdcdc0b4d60 2024-03-25T18:15:42.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.653+0000 7fdcd8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:42.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.653+0000 7fdcd8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:42.654 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:42.654 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.655+0000 7fdce1ee4640 10 librados: watch_flush enter 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.655+0000 7fdce1ee4640 10 librados: watch_flush exit 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.655+0000 7fdce1ee4640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.655+0000 7fdce1ee4640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 >> v1:172.21.15.161:6800/411950978 conn(0x7fdcbc06df20 legacy=0x7fdcbc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 10 monclient: shutdown 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 >> v1:172.21.15.132:6790/0 conn(0x7fdcdc0b4d60 legacy=0x7fdcdc12c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 shutdown_connections 2024-03-25T18:15:42.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 >> 172.21.15.132:0/3004940090 conn(0x7fdcdc0617a0 msgr2=0x7fdcdc0b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:42.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 shutdown_connections 2024-03-25T18:15:42.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 -- 172.21.15.132:0/3004940090 wait complete. 2024-03-25T18:15:42.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:42.656+0000 7fdce1ee4640 1 librados: shutdown 2024-03-25T18:15:42.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:49.478 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:49.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.596+0000 7fb44f6f4640 10 monclient: get_monmap_and_config 2024-03-25T18:15:49.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.596+0000 7fb44f6f4640 10 monclient: build_initial_monmap 2024-03-25T18:15:49.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.596+0000 7fb44f6f4640 10 monclient: monmap: 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:49.597719+0000 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:49.597719+0000 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:49.598 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:49.599 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:49.599 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:49.599 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:49.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.597+0000 7fb44f6f4640 1 Processor -- start 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.597+0000 7fb44f6f4640 1 -- start start 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.597+0000 7fb44f6f4640 10 monclient: init 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient(hunting): picked mon.noname-a con 0x7fb4480a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient(hunting): picked mon.noname-b con 0x7fb44813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient(hunting): picked mon.noname-c con 0x7fb4481430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb448142520 con 0x7fb4480a98f0 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480b28a0 con 0x7fb4481430c0 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480b2a30 con 0x7fb44813f710 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:49.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.598+0000 7fb44f6f4640 10 monclient(hunting): authenticate will time out at 7389.861128s 2024-03-25T18:15:49.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.605+0000 7fb44dc91640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb4481430c0 0x7fb44814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:51098/0 (socket says 172.21.15.132:51098) 2024-03-25T18:15:49.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.605+0000 7fb44dc91640 1 -- 172.21.15.132:0/946595848 learned_addr learned my addr 172.21.15.132:0/946595848 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:49.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.613+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95622884 0 0) 0x7fb4480b2a30 con 0x7fb44813f710 2024-03-25T18:15:49.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.613+0000 7fb43ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.613+0000 7fb43ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.613+0000 7fb43ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.613+0000 7fb43ffff640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:15:49.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.613+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb434003520 con 0x7fb44813f710 2024-03-25T18:15:49.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.615+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616921769 0 0) 0x7fb4480b28a0 con 0x7fb4481430c0 2024-03-25T18:15:49.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.615+0000 7fb43ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.615+0000 7fb43ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.615+0000 7fb43ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.615+0000 7fb43ffff640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:15:49.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.615+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4480b2a30 con 0x7fb4481430c0 2024-03-25T18:15:49.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.622+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073405839 0 0) 0x7fb448142520 con 0x7fb4480a98f0 2024-03-25T18:15:49.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.622+0000 7fb43ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.622+0000 7fb43ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.622+0000 7fb43ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.622+0000 7fb43ffff640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:15:49.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.622+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4480b28a0 con 0x7fb4480a98f0 2024-03-25T18:15:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127796725 0 0) 0x7fb434003520 con 0x7fb44813f710 2024-03-25T18:15:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 >> v1:172.21.15.161:6789/0 conn(0x7fb4481430c0 legacy=0x7fb44814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 >> v1:172.21.15.132:6789/0 conn(0x7fb4480a98f0 legacy=0x7fb4480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 1 monclient: found mon.noname-b 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480b44c0 con 0x7fb44813f710 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: _finish_auth 0 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb438001690 con 0x7fb44813f710 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb43ffff640 10 monclient: dump: 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:49.628 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb44f6f4640 5 monclient: authenticate success, global_id 4316 2024-03-25T18:15:49.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.626+0000 7fb44f6f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4380017f0 con 0x7fb44813f710 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb43ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb43ffff640 1 -- 172.21.15.132:0/946595848 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb438001c00 con 0x7fb44813f710 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb44f6f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb44f6f4640 4 set_mon_vals no callback set 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb43ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb43ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.647+0000 7fb43ffff640 10 monclient: dump: 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:49.648 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:49.649 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:49.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 10 monclient: shutdown 2024-03-25T18:15:49.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:49.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 1 -- 172.21.15.132:0/946595848 >> v1:172.21.15.132:6790/0 conn(0x7fb44813f710 legacy=0x7fb4481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 1 -- 172.21.15.132:0/946595848 shutdown_connections 2024-03-25T18:15:49.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 1 -- 172.21.15.132:0/946595848 >> 172.21.15.132:0/946595848 conn(0x7fb4480b7ea0 msgr2=0x7fb4480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:49.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 1 -- 172.21.15.132:0/946595848 shutdown_connections 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.648+0000 7fb44f6f4640 1 -- 172.21.15.132:0/946595848 wait complete. 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 10 monclient: build_initial_monmap 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 10 monclient: monmap: 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:49.652 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 1 librados: starting msgr at 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 1 librados: starting objecter 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 1 Processor -- start 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 1 -- start start 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 1 librados: setting wanted keys 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 1 librados: calling monclient init 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.649+0000 7fb44f6f4640 10 monclient: init 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient(hunting): picked mon.noname-a con 0x7fb4480a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient(hunting): picked mon.noname-c con 0x7fb44813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient(hunting): picked mon.noname-b con 0x7fb4481430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:49.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4480aba00 con 0x7fb4480a98f0 2024-03-25T18:15:49.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4481d7490 con 0x7fb4481430c0 2024-03-25T18:15:49.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4481d8360 con 0x7fb44813f710 2024-03-25T18:15:49.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:49.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.650+0000 7fb44f6f4640 10 monclient(hunting): authenticate will time out at 7389.913113s 2024-03-25T18:15:49.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.657+0000 7fb44dc91640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb4481430c0 0x7fb448133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:51120/0 (socket says 172.21.15.132:51120) 2024-03-25T18:15:49.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.657+0000 7fb44dc91640 1 -- 172.21.15.132:0/296451941 learned_addr learned my addr 172.21.15.132:0/296451941 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:49.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.660+0000 7fb43dffb640 10 client.?.objecter ms_handle_connect 0x7fb4481430c0 2024-03-25T18:15:49.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.660+0000 7fb43dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.661+0000 7fb43dffb640 10 client.?.objecter ms_handle_connect 0x7fb44813f710 2024-03-25T18:15:49.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.661+0000 7fb43dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806464853 0 0) 0x7fb4481d7490 con 0x7fb4481430c0 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb42c004eb0 con 0x7fb4481430c0 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266462018 0 0) 0x7fb4481d8360 con 0x7fb44813f710 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:15:49.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.668+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4481d7490 con 0x7fb44813f710 2024-03-25T18:15:49.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.674+0000 7fb43dffb640 10 client.?.objecter ms_handle_connect 0x7fb4480a98f0 2024-03-25T18:15:49.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.674+0000 7fb43dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704860590 0 0) 0x7fb4481d7490 con 0x7fb44813f710 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4481d8360 con 0x7fb44813f710 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb438001b00 con 0x7fb44813f710 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 10 monclient(hunting): dump: 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.679 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.680 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:49.680 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:49.680 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:49.680 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:49.680 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:49.680 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.678+0000 7fb43dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.684+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3683748501 0 0) 0x7fb42c004eb0 con 0x7fb4481430c0 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.684+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4481d7490 con 0x7fb4481430c0 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.684+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb440001c70 con 0x7fb4481430c0 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.684+0000 7fb43dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.684+0000 7fb43dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.684+0000 7fb43dffb640 10 monclient(hunting): dump: 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:49.686 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:49.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.689+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855841149 0 0) 0x7fb4480aba00 con 0x7fb4480a98f0 2024-03-25T18:15:49.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.690+0000 7fb43dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:49.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.690+0000 7fb43dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:49.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.690+0000 7fb43dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:49.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.690+0000 7fb43dffb640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:15:49.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.690+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb42c004eb0 con 0x7fb4480a98f0 2024-03-25T18:15:49.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 690436006 0 0) 0x7fb4481d8360 con 0x7fb44813f710 2024-03-25T18:15:49.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 >> v1:172.21.15.161:6789/0 conn(0x7fb4481430c0 legacy=0x7fb448133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 >> v1:172.21.15.132:6789/0 conn(0x7fb4480a98f0 legacy=0x7fb4481386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 1 monclient: found mon.c 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4480abc30 con 0x7fb44813f710 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb43dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 5 monclient: authenticate success, global_id 4319 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 10 monclient: _renew_subs 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:49.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb448137940 con 0x7fb44813f710 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 10 monclient: _renew_subs 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb448135220 con 0x7fb44813f710 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.691+0000 7fb44f6f4640 1 librados: init done 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.692+0000 7fb44f6f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.692+0000 7fb44f6f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:49.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.692+0000 7fb44f6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:49.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:49.692+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4480627b0 con 0x7fb44813f710 2024-03-25T18:15:50.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4380038a0 con 0x7fb44813f710 2024-03-25T18:15:50.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:50.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb438003bb0 con 0x7fb44813f710 2024-03-25T18:15:50.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:50.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43dffb640 10 monclient: dump: 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:50.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.001+0000 7fb43f7fe640 4 set_mon_vals no callback set 2024-03-25T18:15:50.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.002+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fb438019b80 con 0x7fb44813f710 2024-03-25T18:15:50.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.002+0000 7fb44d490640 10 monclient: get_auth_request con 0x7fb42c06df90 auth_method 0 2024-03-25T18:15:50.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.003+0000 7fb43dffb640 10 client.4319.objecter ms_handle_connect 0x7fb42c06df90 2024-03-25T18:15:50.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fb438053670 con 0x7fb44813f710 2024-03-25T18:15:50.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 10 client.4319.objecter ms_dispatch 0x7fb448000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:50.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:50.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:50.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-25T18:15:50.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4380b08b0 con 0x7fb44813f710 2024-03-25T18:15:50.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:50.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.007+0000 7fb43dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:50.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.257+0000 7fb44f6f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.257+0000 7fb44f6f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.257+0000 7fb44f6f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:15:50.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.257+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb448069f40 con 0x7fb44813f710 2024-03-25T18:15:50.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.315+0000 7fb43dffb640 1 -- 172.21.15.132:0/296451941 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fb43804efb0 con 0x7fb44813f710 2024-03-25T18:15:50.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.315+0000 7fb43dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.315+0000 7fb43dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:50.316 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:50.316 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 10 librados: watch_flush enter 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 10 librados: watch_flush exit 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 10 client.4319.objecter successfully canceled tick 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 >> v1:172.21.15.161:6800/411950978 conn(0x7fb42c06df90 legacy=0x7fb42c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 10 monclient: shutdown 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 >> v1:172.21.15.132:6790/0 conn(0x7fb44813f710 legacy=0x7fb4481d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 shutdown_connections 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 >> 172.21.15.132:0/296451941 conn(0x7fb4480b7ea0 msgr2=0x7fb448074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:50.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 shutdown_connections 2024-03-25T18:15:50.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 -- 172.21.15.132:0/296451941 wait complete. 2024-03-25T18:15:50.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:50.317+0000 7fb44f6f4640 1 librados: shutdown 2024-03-25T18:15:50.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:54.345 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:39.609 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.348 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:39.612 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.381 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:39.645 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.416 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:39.658 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.423 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:39.687 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.424 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:39.666 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.427 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:39.691 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.453 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:39.695 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:54.461 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:39.725 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:57.135 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:57.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.251+0000 7f2ddddb6640 10 monclient: get_monmap_and_config 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.251+0000 7f2ddddb6640 10 monclient: build_initial_monmap 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.251+0000 7f2ddddb6640 10 monclient: monmap: 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:15:57.252144+0000 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:15:57.252144+0000 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:15:57.252 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.251+0000 7f2ddddb6640 1 Processor -- start 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.251+0000 7f2ddddb6640 1 -- start start 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.251+0000 7f2ddddb6640 10 monclient: init 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd80a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd81430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd8142520 con 0x7f2dd813f710 2024-03-25T18:15:57.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b28a0 con 0x7f2dd80a98f0 2024-03-25T18:15:57.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80b2a30 con 0x7f2dd81430c0 2024-03-25T18:15:57.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:57.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.252+0000 7f2ddddb6640 10 monclient(hunting): authenticate will time out at 7397.515241s 2024-03-25T18:15:57.261 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.260+0000 7f2dd77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2dd80a98f0 0x7f2dd80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:56692/0 (socket says 172.21.15.132:56692) 2024-03-25T18:15:57.261 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.260+0000 7f2dd77fe640 1 -- 172.21.15.132:0/2284981742 learned_addr learned my addr 172.21.15.132:0/2284981742 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:57.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.266+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028206866 0 0) 0x7f2dd80b28a0 con 0x7f2dd80a98f0 2024-03-25T18:15:57.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.266+0000 7f2dd67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.266+0000 7f2dd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.266+0000 7f2dd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.266+0000 7f2dd67fc640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:15:57.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.266+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc0003520 con 0x7f2dd80a98f0 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3977584466 0 0) 0x7f2dc0003520 con 0x7f2dd80a98f0 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 >> v1:172.21.15.132:6790/0 conn(0x7f2dd81430c0 legacy=0x7f2dd814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 >> v1:172.21.15.132:6789/0 conn(0x7f2dd813f710 legacy=0x7f2dd81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd80b44c0 con 0x7f2dd80a98f0 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:57.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc8001690 con 0x7f2dd80a98f0 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2dd67fc640 10 monclient: dump: 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2ddddb6640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:15:57.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.275+0000 7f2ddddb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc8001870 con 0x7f2dd80a98f0 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2dd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2dd67fc640 1 -- 172.21.15.132:0/2284981742 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc8001c80 con 0x7f2dd80a98f0 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2ddddb6640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2ddddb6640 4 set_mon_vals no callback set 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2dd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2dd67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.294+0000 7f2dd67fc640 10 monclient: dump: 2024-03-25T18:15:57.295 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:57.296 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.297 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 10 monclient: shutdown 2024-03-25T18:15:57.297 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 1 -- 172.21.15.132:0/2284981742 >> v1:172.21.15.161:6789/0 conn(0x7f2dd80a98f0 legacy=0x7f2dd80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 1 -- 172.21.15.132:0/2284981742 shutdown_connections 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 1 -- 172.21.15.132:0/2284981742 >> 172.21.15.132:0/2284981742 conn(0x7f2dd80b7ea0 msgr2=0x7f2dd80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 1 -- 172.21.15.132:0/2284981742 shutdown_connections 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.295+0000 7f2ddddb6640 1 -- 172.21.15.132:0/2284981742 wait complete. 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.296+0000 7f2ddddb6640 10 monclient: build_initial_monmap 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.296+0000 7f2ddddb6640 10 monclient: monmap: 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:15:57.298 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.296+0000 7f2ddddb6640 1 librados: starting msgr at 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.296+0000 7f2ddddb6640 1 librados: starting objecter 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.296+0000 7f2ddddb6640 1 Processor -- start 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.297+0000 7f2ddddb6640 1 -- start start 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.297+0000 7f2ddddb6640 1 librados: setting wanted keys 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.297+0000 7f2ddddb6640 1 librados: calling monclient init 2024-03-25T18:15:57.299 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.297+0000 7f2ddddb6640 10 monclient: init 2024-03-25T18:15:57.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:57.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:57.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient(hunting): picked mon.noname-b con 0x7f2dd80a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:15:57.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient(hunting): picked mon.noname-a con 0x7f2dd813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:15:57.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient(hunting): picked mon.noname-c con 0x7f2dd81430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:15:57.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd80ab9e0 con 0x7f2dd813f710 2024-03-25T18:15:57.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd8138c20 con 0x7f2dd80a98f0 2024-03-25T18:15:57.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dd8138f00 con 0x7f2dd81430c0 2024-03-25T18:15:57.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:57.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.298+0000 7f2ddddb6640 10 monclient(hunting): authenticate will time out at 7397.561233s 2024-03-25T18:15:57.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.313+0000 7f2dd7fff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f2dd81430c0 0x7f2dd81e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:34990/0 (socket says 172.21.15.132:34990) 2024-03-25T18:15:57.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.313+0000 7f2dd7fff640 1 -- 172.21.15.132:0/3999618817 learned_addr learned my addr 172.21.15.132:0/3999618817 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:15:57.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.314+0000 7f2dbffff640 10 client.?.objecter ms_handle_connect 0x7f2dd80a98f0 2024-03-25T18:15:57.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.314+0000 7f2dbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.316+0000 7f2dbffff640 10 client.?.objecter ms_handle_connect 0x7f2dd81430c0 2024-03-25T18:15:57.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.316+0000 7f2dbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.319+0000 7f2dbffff640 10 client.?.objecter ms_handle_connect 0x7f2dd813f710 2024-03-25T18:15:57.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.319+0000 7f2dbffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.324+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061593380 0 0) 0x7f2dd80ab9e0 con 0x7f2dd813f710 2024-03-25T18:15:57.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.324+0000 7f2dbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.324+0000 7f2dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.324+0000 7f2dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.324+0000 7f2dbffff640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:15:57.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.324+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc4004eb0 con 0x7f2dd813f710 2024-03-25T18:15:57.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.326+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3470204307 0 0) 0x7f2dd8138f00 con 0x7f2dd81430c0 2024-03-25T18:15:57.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.326+0000 7f2dbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.326+0000 7f2dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.326+0000 7f2dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.326+0000 7f2dbffff640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:15:57.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.326+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd80ab9e0 con 0x7f2dd81430c0 2024-03-25T18:15:57.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.331+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549806220 0 0) 0x7f2dd8138c20 con 0x7f2dd80a98f0 2024-03-25T18:15:57.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.331+0000 7f2dbffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.331+0000 7f2dbffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.331+0000 7f2dbffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.331+0000 7f2dbffff640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:15:57.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.331+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd8138f00 con 0x7f2dd80a98f0 2024-03-25T18:15:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.336+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 259779054 0 0) 0x7f2dc4004eb0 con 0x7f2dd813f710 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.337+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd8138c20 con 0x7f2dd813f710 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.337+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dcc0018c0 con 0x7f2dd813f710 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.337+0000 7f2dbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.337+0000 7f2dbffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.337+0000 7f2dbffff640 10 monclient(hunting): dump: 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.338 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.339 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:57.339 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:57.339 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:57.339 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.337+0000 7f2dbffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.340+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2047358480 0 0) 0x7f2dd80ab9e0 con 0x7f2dd81430c0 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.340+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc4004eb0 con 0x7f2dd81430c0 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.340+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dd0001c10 con 0x7f2dd81430c0 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.340+0000 7f2dbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.340+0000 7f2dbffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.340+0000 7f2dbffff640 10 monclient(hunting): dump: 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.341 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:57.342 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:57.342 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:57.342 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.341+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4170270057 0 0) 0x7f2dd8138f00 con 0x7f2dd80a98f0 2024-03-25T18:15:57.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.341+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dd80ab9e0 con 0x7f2dd80a98f0 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.342+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc8001b80 con 0x7f2dd80a98f0 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.342+0000 7f2dbffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.342+0000 7f2dbffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.342+0000 7f2dbffff640 10 monclient(hunting): dump: 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:57.343 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 296147257 0 0) 0x7f2dd8138c20 con 0x7f2dd813f710 2024-03-25T18:15:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 >> v1:172.21.15.132:6790/0 conn(0x7f2dd81430c0 legacy=0x7f2dd81e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 >> v1:172.21.15.161:6789/0 conn(0x7f2dd80a98f0 legacy=0x7f2dd8137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 1 monclient: found mon.a 2024-03-25T18:15:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd814e7a0 con 0x7f2dd813f710 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 10 monclient: _finish_auth 0 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2dbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 10 monclient: _renew_subs 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dd81386d0 con 0x7f2dd813f710 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 10 monclient: _renew_subs 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dd81d8060 con 0x7f2dd813f710 2024-03-25T18:15:57.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.345+0000 7f2ddddb6640 1 librados: init done 2024-03-25T18:15:57.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.346+0000 7f2ddddb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.346+0000 7f2ddddb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.346+0000 7f2ddddb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:57.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.346+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dd8074cc0 con 0x7f2dd813f710 2024-03-25T18:15:57.647 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dcc0035f0 con 0x7f2dd813f710 2024-03-25T18:15:57.647 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dbffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:57.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dcc003900 con 0x7f2dd813f710 2024-03-25T18:15:57.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dbffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:57.648 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dbffff640 10 monclient: dump: 2024-03-25T18:15:57.648 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:15:57.648 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:15:57.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.646+0000 7f2dd5ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:57.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.647+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f2dcc0198d0 con 0x7f2dd813f710 2024-03-25T18:15:57.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.648+0000 7f2dd77fe640 10 monclient: get_auth_request con 0x7f2dc406e0b0 auth_method 0 2024-03-25T18:15:57.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.648+0000 7f2dbffff640 10 client.4332.objecter ms_handle_connect 0x7f2dc406e0b0 2024-03-25T18:15:57.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f2dcc0535a0 con 0x7f2dd813f710 2024-03-25T18:15:57.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 10 client.4332.objecter ms_dispatch 0x7f2dd8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:57.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:57.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:57.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:15:57.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dcc04eee0 con 0x7f2dd813f710 2024-03-25T18:15:57.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.654+0000 7f2dbffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:57.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.873+0000 7f2ddddb6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.873+0000 7f2ddddb6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.873+0000 7f2ddddb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:15:57.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.873+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dd80627b0 con 0x7f2dd813f710 2024-03-25T18:15:57.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.939+0000 7f2dbffff640 1 -- 172.21.15.132:0/3999618817 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f2dcc050fe0 con 0x7f2dd813f710 2024-03-25T18:15:57.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.939+0000 7f2dbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:57.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.939+0000 7f2dbffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:57.940 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:15:57.940 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 10 librados: watch_flush enter 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 10 librados: watch_flush exit 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 >> v1:172.21.15.161:6800/411950978 conn(0x7f2dc406e0b0 legacy=0x7f2dc4070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 10 monclient: shutdown 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 >> v1:172.21.15.132:6789/0 conn(0x7f2dd813f710 legacy=0x7f2dd81d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 shutdown_connections 2024-03-25T18:15:57.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 >> 172.21.15.132:0/3999618817 conn(0x7f2dd80b7ea0 msgr2=0x7f2dd8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:57.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.941+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 shutdown_connections 2024-03-25T18:15:57.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.942+0000 7f2ddddb6640 1 -- 172.21.15.132:0/3999618817 wait complete. 2024-03-25T18:15:57.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:15:57.942+0000 7f2ddddb6640 1 librados: shutdown 2024-03-25T18:15:57.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:04.761 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:04.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.875+0000 7f49edb96640 10 monclient: get_monmap_and_config 2024-03-25T18:16:04.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.875+0000 7f49edb96640 10 monclient: build_initial_monmap 2024-03-25T18:16:04.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.875+0000 7f49edb96640 10 monclient: monmap: 2024-03-25T18:16:04.876 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:04.876 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:04.876 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:04.876641+0000 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:04.876641+0000 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:04.877 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.876+0000 7f49edb96640 1 Processor -- start 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.876+0000 7f49edb96640 1 -- start start 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.876+0000 7f49edb96640 10 monclient: init 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient(hunting): picked mon.noname-c con 0x7f49e80a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient(hunting): picked mon.noname-b con 0x7f49e813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient(hunting): picked mon.noname-a con 0x7f49e81430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:04.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e8142520 con 0x7f49e81430c0 2024-03-25T18:16:04.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e80b28a0 con 0x7f49e80a98f0 2024-03-25T18:16:04.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e80b2a30 con 0x7f49e813f710 2024-03-25T18:16:04.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:04.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.877+0000 7f49edb96640 10 monclient(hunting): authenticate will time out at 7405.140096s 2024-03-25T18:16:04.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.892+0000 7f49e6ffd640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f49e813f710 0x7f49e81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:48508/0 (socket says 172.21.15.132:48508) 2024-03-25T18:16:04.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.892+0000 7f49e6ffd640 1 -- 172.21.15.132:0/199707572 learned_addr learned my addr 172.21.15.132:0/199707572 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:04.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.908+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645810868 0 0) 0x7f49e80b2a30 con 0x7f49e813f710 2024-03-25T18:16:04.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.908+0000 7f49e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.908+0000 7f49e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.908+0000 7f49e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.908+0000 7f49e67fc640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:16:04.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.908+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d0003520 con 0x7f49e813f710 2024-03-25T18:16:04.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.910+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955487495 0 0) 0x7f49e80b28a0 con 0x7f49e80a98f0 2024-03-25T18:16:04.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.910+0000 7f49e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.910+0000 7f49e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.910+0000 7f49e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.910+0000 7f49e67fc640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:16:04.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.910+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e80b2a30 con 0x7f49e80a98f0 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797868145 0 0) 0x7f49d0003520 con 0x7f49e813f710 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 >> v1:172.21.15.161:6789/0 conn(0x7f49e80a98f0 legacy=0x7f49e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 >> v1:172.21.15.132:6789/0 conn(0x7f49e81430c0 legacy=0x7f49e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 1 monclient: found mon.noname-b 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e80b44c0 con 0x7f49e813f710 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f49dc001690 con 0x7f49e813f710 2024-03-25T18:16:04.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49edb96640 5 monclient: authenticate success, global_id 4328 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49edb96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49e67fc640 10 monclient: dump: 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:04.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.916+0000 7f49edb96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49dc0017f0 con 0x7f49e813f710 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49e67fc640 1 -- 172.21.15.132:0/199707572 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f49dc001c00 con 0x7f49e813f710 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49e67fc640 10 monclient: dump: 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:04.951 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49edb96640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:04.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.950+0000 7f49edb96640 4 set_mon_vals no callback set 2024-03-25T18:16:04.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 10 monclient: shutdown 2024-03-25T18:16:04.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:04.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 1 -- 172.21.15.132:0/199707572 >> v1:172.21.15.132:6790/0 conn(0x7f49e813f710 legacy=0x7f49e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 1 -- 172.21.15.132:0/199707572 shutdown_connections 2024-03-25T18:16:04.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 1 -- 172.21.15.132:0/199707572 >> 172.21.15.132:0/199707572 conn(0x7f49e80b7ea0 msgr2=0x7f49e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 1 -- 172.21.15.132:0/199707572 shutdown_connections 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.951+0000 7f49edb96640 1 -- 172.21.15.132:0/199707572 wait complete. 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 10 monclient: build_initial_monmap 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 10 monclient: monmap: 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:04.954 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 1 librados: starting msgr at 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 1 librados: starting objecter 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 1 Processor -- start 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 1 -- start start 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 1 librados: setting wanted keys 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 1 librados: calling monclient init 2024-03-25T18:16:04.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.952+0000 7f49edb96640 10 monclient: init 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient(hunting): picked mon.noname-b con 0x7f49e80a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient(hunting): picked mon.noname-c con 0x7f49e813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient(hunting): picked mon.noname-a con 0x7f49e81430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e81370a0 con 0x7f49e81430c0 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e8135090 con 0x7f49e80a98f0 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e8135220 con 0x7f49e813f710 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:04.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.953+0000 7f49edb96640 10 monclient(hunting): authenticate will time out at 7405.216488s 2024-03-25T18:16:04.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.957+0000 7f49e7fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f49e81430c0 0x7f49e81d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:36778/0 (socket says 172.21.15.132:36778) 2024-03-25T18:16:04.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.957+0000 7f49e7fff640 1 -- 172.21.15.132:0/1090537706 learned_addr learned my addr 172.21.15.132:0/1090537706 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:04.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.962+0000 7f49cffff640 10 client.?.objecter ms_handle_connect 0x7f49e80a98f0 2024-03-25T18:16:04.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.962+0000 7f49cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.962+0000 7f49cffff640 10 client.?.objecter ms_handle_connect 0x7f49e81430c0 2024-03-25T18:16:04.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.962+0000 7f49cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.967+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3511134499 0 0) 0x7f49e81370a0 con 0x7f49e81430c0 2024-03-25T18:16:04.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.967+0000 7f49cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.967+0000 7f49cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.967+0000 7f49cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.967+0000 7f49cffff640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:16:04.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.967+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49d4004eb0 con 0x7f49e81430c0 2024-03-25T18:16:04.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.970+0000 7f49cffff640 10 client.?.objecter ms_handle_connect 0x7f49e813f710 2024-03-25T18:16:04.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.970+0000 7f49cffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:04.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.972+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411145828 0 0) 0x7f49e8135090 con 0x7f49e80a98f0 2024-03-25T18:16:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.972+0000 7f49cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.972+0000 7f49cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.972+0000 7f49cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.972+0000 7f49cffff640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:16:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.972+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e81370a0 con 0x7f49e80a98f0 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700732952 0 0) 0x7f49d4004eb0 con 0x7f49e81430c0 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49e8135090 con 0x7f49e81430c0 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f49e0001c30 con 0x7f49e81430c0 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 10 monclient(hunting): dump: 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.978 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:04.979 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:04.979 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:04.979 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:04.979 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:04.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.977+0000 7f49cffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2561677874 0 0) 0x7f49e81370a0 con 0x7f49e80a98f0 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49d4004eb0 con 0x7f49e80a98f0 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f49d8001930 con 0x7f49e80a98f0 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): dump: 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:04.985 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056484715 0 0) 0x7f49e8135220 con 0x7f49e813f710 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:16:04.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.984+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e81370a0 con 0x7f49e813f710 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2582208000 0 0) 0x7f49d4004eb0 con 0x7f49e80a98f0 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 >> v1:172.21.15.132:6790/0 conn(0x7f49e813f710 legacy=0x7f49e81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 >> v1:172.21.15.132:6789/0 conn(0x7f49e81430c0 legacy=0x7f49e81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 1 monclient: found mon.b 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e81d5000 con 0x7f49e80a98f0 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 10 monclient: _finish_auth 0 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 10 monclient: _renew_subs 2024-03-25T18:16:04.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e81d52b0 con 0x7f49e80a98f0 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 10 monclient: _renew_subs 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e8135910 con 0x7f49e80a98f0 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.994+0000 7f49edb96640 1 librados: init done 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.995+0000 7f49edb96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.995+0000 7f49edb96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:04.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.995+0000 7f49edb96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:04.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:04.995+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e80627b0 con 0x7f49e80a98f0 2024-03-25T18:16:05.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49d8003610 con 0x7f49e80a98f0 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49e5ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f49d80039a0 con 0x7f49e80a98f0 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.264+0000 7f49cffff640 10 monclient: dump: 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.265+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f49d8003bc0 con 0x7f49e80a98f0 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.265+0000 7f49e6ffd640 10 monclient: get_auth_request con 0x7f49d406df50 auth_method 0 2024-03-25T18:16:05.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.266+0000 7f49cffff640 10 client.4336.objecter ms_handle_connect 0x7f49d406df50 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.269+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f49d8053590 con 0x7f49e80a98f0 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.269+0000 7f49cffff640 10 client.4336.objecter ms_dispatch 0x7f49e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.269+0000 7f49cffff640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.269+0000 7f49cffff640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.270+0000 7f49cffff640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.270+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49d804f0a0 con 0x7f49e80a98f0 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.270+0000 7f49cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.270+0000 7f49cffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:05.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.485+0000 7f49edb96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.485+0000 7f49edb96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.485+0000 7f49edb96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:05.487 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.485+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49e813bfe0 con 0x7f49e80a98f0 2024-03-25T18:16:05.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.507+0000 7f49cffff640 1 -- 172.21.15.132:0/1090537706 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f49d8050fd0 con 0x7f49e80a98f0 2024-03-25T18:16:05.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.507+0000 7f49cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.507+0000 7f49cffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:05.508 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:05.508 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:05.511 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 10 librados: watch_flush enter 2024-03-25T18:16:05.511 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 10 librados: watch_flush exit 2024-03-25T18:16:05.511 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 >> v1:172.21.15.161:6800/411950978 conn(0x7f49d406df50 legacy=0x7f49d4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 10 monclient: shutdown 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 >> v1:172.21.15.161:6789/0 conn(0x7f49e80a98f0 legacy=0x7f49e81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 shutdown_connections 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 >> 172.21.15.132:0/1090537706 conn(0x7f49e80b7ea0 msgr2=0x7f49e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 shutdown_connections 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 -- 172.21.15.132:0/1090537706 wait complete. 2024-03-25T18:16:05.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:05.510+0000 7f49edb96640 1 librados: shutdown 2024-03-25T18:16:05.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:12.328 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:12.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.442+0000 7f79a9c1e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:12.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.442+0000 7f79a9c1e640 10 monclient: build_initial_monmap 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.442+0000 7f79a9c1e640 10 monclient: monmap: 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:12.443915+0000 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:12.443915+0000 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:12.444 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.443+0000 7f79a9c1e640 1 Processor -- start 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 1 -- start start 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient: init 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient(hunting): picked mon.noname-a con 0x7f79a406adf0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient(hunting): picked mon.noname-b con 0x7f79a40aee10 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient(hunting): picked mon.noname-c con 0x7f79a4140860 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a40af690 con 0x7f79a406adf0 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a40b0c70 con 0x7f79a4140860 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a40b0f50 con 0x7f79a40aee10 2024-03-25T18:16:12.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.444+0000 7f79a9c1e640 10 monclient(hunting): authenticate will time out at 7412.707570s 2024-03-25T18:16:12.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.447+0000 7f79a3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f79a4140860 0x7f79a414b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:58336/0 (socket says 172.21.15.132:58336) 2024-03-25T18:16:12.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.447+0000 7f79a3fff640 1 -- 172.21.15.132:0/198506960 learned_addr learned my addr 172.21.15.132:0/198506960 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:12.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.454+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761056000 0 0) 0x7f79a40b0c70 con 0x7f79a4140860 2024-03-25T18:16:12.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.454+0000 7f79a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.454+0000 7f79a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.454+0000 7f79a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.454+0000 7f79a2ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:16:12.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.454+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7990003520 con 0x7f79a4140860 2024-03-25T18:16:12.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411822795 0 0) 0x7f79a40b0f50 con 0x7f79a40aee10 2024-03-25T18:16:12.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:16:12.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a40b0c70 con 0x7f79a40aee10 2024-03-25T18:16:12.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120812150 0 0) 0x7f79a40af690 con 0x7f79a406adf0 2024-03-25T18:16:12.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:16:12.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.465+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a40b0f50 con 0x7f79a406adf0 2024-03-25T18:16:12.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435573931 0 0) 0x7f7990003520 con 0x7f79a4140860 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 >> v1:172.21.15.132:6790/0 conn(0x7f79a40aee10 legacy=0x7f79a4140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 >> v1:172.21.15.132:6789/0 conn(0x7f79a406adf0 legacy=0x7f79a413d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a413dc40 con 0x7f79a4140860 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f799c001a10 con 0x7f79a4140860 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a2ffd640 10 monclient: dump: 2024-03-25T18:16:12.469 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a9c1e640 5 monclient: authenticate success, global_id 4339 2024-03-25T18:16:12.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.467+0000 7f79a9c1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f799c001b70 con 0x7f79a4140860 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a2ffd640 1 -- 172.21.15.132:0/198506960 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f799c001f80 con 0x7f79a4140860 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a2ffd640 10 monclient: dump: 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:12.497 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a9c1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:12.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.496+0000 7f79a9c1e640 4 set_mon_vals no callback set 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 10 monclient: shutdown 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 1 -- 172.21.15.132:0/198506960 >> v1:172.21.15.161:6789/0 conn(0x7f79a4140860 legacy=0x7f79a414b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 1 -- 172.21.15.132:0/198506960 shutdown_connections 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 1 -- 172.21.15.132:0/198506960 >> 172.21.15.132:0/198506960 conn(0x7f79a40b4a80 msgr2=0x7f79a40b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 1 -- 172.21.15.132:0/198506960 shutdown_connections 2024-03-25T18:16:12.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.497+0000 7f79a9c1e640 1 -- 172.21.15.132:0/198506960 wait complete. 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.498+0000 7f79a9c1e640 10 monclient: build_initial_monmap 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.498+0000 7f79a9c1e640 10 monclient: monmap: 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:12.500 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.498+0000 7f79a9c1e640 1 librados: starting msgr at 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.498+0000 7f79a9c1e640 1 librados: starting objecter 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.498+0000 7f79a9c1e640 1 Processor -- start 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 1 -- start start 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 1 librados: setting wanted keys 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 1 librados: calling monclient init 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient: init 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:12.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient(hunting): picked mon.noname-a con 0x7f79a406adf0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient(hunting): picked mon.noname-b con 0x7f79a40aee10 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient(hunting): picked mon.noname-c con 0x7f79a4070740 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a413afc0 con 0x7f79a406adf0 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a41399b0 con 0x7f79a40aee10 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79a4138e00 con 0x7f79a4070740 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.499+0000 7f79a9c1e640 10 monclient(hunting): authenticate will time out at 7412.762551s 2024-03-25T18:16:12.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.504+0000 7f79a3fff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f79a4070740 0x7f79a41e7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:53270/0 (socket says 172.21.15.132:53270) 2024-03-25T18:16:12.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.504+0000 7f79a3fff640 1 -- 172.21.15.132:0/2195698517 learned_addr learned my addr 172.21.15.132:0/2195698517 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:12.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.508+0000 7f79a0ff9640 10 client.?.objecter ms_handle_connect 0x7f79a4070740 2024-03-25T18:16:12.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.508+0000 7f79a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.508+0000 7f79a0ff9640 10 client.?.objecter ms_handle_connect 0x7f79a40aee10 2024-03-25T18:16:12.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.508+0000 7f79a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261029535 0 0) 0x7f79a4138e00 con 0x7f79a4070740 2024-03-25T18:16:12.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:16:12.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7984004eb0 con 0x7f79a4070740 2024-03-25T18:16:12.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.514+0000 7f79a0ff9640 10 client.?.objecter ms_handle_connect 0x7f79a406adf0 2024-03-25T18:16:12.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.515+0000 7f79a0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.520+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517275275 0 0) 0x7f79a41399b0 con 0x7f79a40aee10 2024-03-25T18:16:12.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.520+0000 7f79a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.520+0000 7f79a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.520+0000 7f79a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.520+0000 7f79a0ff9640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:16:12.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.520+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a4138e00 con 0x7f79a40aee10 2024-03-25T18:16:12.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.521+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220657205 0 0) 0x7f79a413afc0 con 0x7f79a406adf0 2024-03-25T18:16:12.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.521+0000 7f79a0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.521+0000 7f79a0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.521+0000 7f79a0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.521+0000 7f79a0ff9640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:16:12.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.521+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a41399b0 con 0x7f79a406adf0 2024-03-25T18:16:12.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2825905218 0 0) 0x7f7984004eb0 con 0x7f79a4070740 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79a413afc0 con 0x7f79a4070740 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f799c001ec0 con 0x7f79a4070740 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:12.525 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:12.526 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:12.526 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:12.526 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:12.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.524+0000 7f79a0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:12.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3070812970 0 0) 0x7f79a413afc0 con 0x7f79a4070740 2024-03-25T18:16:12.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 >> v1:172.21.15.161:6789/0 conn(0x7f79a40aee10 legacy=0x7f79a4136bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 >> v1:172.21.15.132:6789/0 conn(0x7f79a406adf0 legacy=0x7f79a41deeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 1 monclient: found mon.c 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79a4138f90 con 0x7f79a4070740 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 5 monclient: authenticate success, global_id 4337 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 10 monclient: _renew_subs 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79a413cf00 con 0x7f79a4070740 2024-03-25T18:16:12.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 10 monclient: _renew_subs 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79a41372d0 con 0x7f79a4070740 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.532+0000 7f79a9c1e640 1 librados: init done 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.533+0000 7f79a9c1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.533+0000 7f79a9c1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.533+0000 7f79a9c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:12.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.533+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79a413b150 con 0x7f79a4070740 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f799c004e10 con 0x7f79a4070740 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f799c0015d0 con 0x7f79a4070740 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a0ff9640 10 monclient: dump: 2024-03-25T18:16:12.835 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:12.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.834+0000 7f79a27fc640 4 set_mon_vals no callback set 2024-03-25T18:16:12.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.835+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f799c01af90 con 0x7f79a4070740 2024-03-25T18:16:12.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.835+0000 7f79a37fe640 10 monclient: get_auth_request con 0x7f798406dda0 auth_method 0 2024-03-25T18:16:12.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.835+0000 7f79a0ff9640 10 client.4337.objecter ms_handle_connect 0x7f798406dda0 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f799c054b10 con 0x7f79a4070740 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 10 client.4337.objecter ms_dispatch 0x7f79a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f799c050620 con 0x7f79a4070740 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:12.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:12.840+0000 7f79a0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:13.094 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.093+0000 7f79a9c1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.094 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.093+0000 7f79a9c1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.094 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.093+0000 7f79a9c1e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:13.094 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.093+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79a406b1d0 con 0x7f79a4070740 2024-03-25T18:16:13.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.135+0000 7f79a0ff9640 1 -- 172.21.15.132:0/2195698517 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f799c052550 con 0x7f79a4070740 2024-03-25T18:16:13.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.135+0000 7f79a0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.135+0000 7f79a0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:13.136 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:13.136 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:13.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 10 librados: watch_flush enter 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 10 librados: watch_flush exit 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 10 client.4337.objecter successfully canceled tick 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 >> v1:172.21.15.161:6800/411950978 conn(0x7f798406dda0 legacy=0x7f7984070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 10 monclient: shutdown 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 >> v1:172.21.15.132:6790/0 conn(0x7f79a4070740 legacy=0x7f79a41e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 shutdown_connections 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 >> 172.21.15.132:0/2195698517 conn(0x7f79a40b4a80 msgr2=0x7f79a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 shutdown_connections 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 -- 172.21.15.132:0/2195698517 wait complete. 2024-03-25T18:16:13.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:13.137+0000 7f79a9c1e640 1 librados: shutdown 2024-03-25T18:16:13.151 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:14.400 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:59.664 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.410 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:59.674 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.459 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:59.701 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.460 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:02:59.725 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.473 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:59.715 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.475 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:59.739 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.481 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:59.745 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.494 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:02:59.736 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.506 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:02:59.770 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:19.957 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:20.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.068+0000 7f588eea8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:20.069 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.068+0000 7f588eea8640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.068+0000 7f588eea8640 10 monclient: monmap: 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:20.069849+0000 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:20.069849+0000 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:20.070 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.069+0000 7f588eea8640 1 Processor -- start 2024-03-25T18:16:20.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.069+0000 7f588eea8640 1 -- start start 2024-03-25T18:16:20.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.069+0000 7f588eea8640 10 monclient: init 2024-03-25T18:16:20.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:20.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient(hunting): picked mon.noname-a con 0x7f58880a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient(hunting): picked mon.noname-c con 0x7f588813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient(hunting): picked mon.noname-b con 0x7f58881430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5888142520 con 0x7f58880a98f0 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58880b28a0 con 0x7f588813f710 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58880b2a30 con 0x7f58881430c0 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.070+0000 7f588eea8640 10 monclient(hunting): authenticate will time out at 7420.333337s 2024-03-25T18:16:20.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.073+0000 7f588cc44640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f58880a98f0 0x7f58880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:48282/0 (socket says 172.21.15.132:48282) 2024-03-25T18:16:20.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.073+0000 7f588cc44640 1 -- 172.21.15.132:0/2847425598 learned_addr learned my addr 172.21.15.132:0/2847425598 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:20.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.084+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239149752 0 0) 0x7f5888142520 con 0x7f58880a98f0 2024-03-25T18:16:20.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.084+0000 7f58877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.084+0000 7f58877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.084+0000 7f58877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.084+0000 7f58877fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:16:20.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.084+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878003520 con 0x7f58880a98f0 2024-03-25T18:16:20.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2742698907 0 0) 0x7f5878003520 con 0x7f58880a98f0 2024-03-25T18:16:20.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 >> v1:172.21.15.132:6790/0 conn(0x7f58881430c0 legacy=0x7f588814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 >> v1:172.21.15.161:6789/0 conn(0x7f588813f710 legacy=0x7f58881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.097 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 1 monclient: found mon.noname-a 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58880b44c0 con 0x7f58880a98f0 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f587c001690 con 0x7f58880a98f0 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f588eea8640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f588eea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f58877fe640 10 monclient: dump: 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:20.098 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.096+0000 7f588eea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f587c0017f0 con 0x7f58880a98f0 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f58877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f58877fe640 1 -- 172.21.15.132:0/2847425598 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f587c001c00 con 0x7f58880a98f0 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f588eea8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f588eea8640 4 set_mon_vals no callback set 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f58877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f58877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.107+0000 7f58877fe640 10 monclient: dump: 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:20.108 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:20.109 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 10 monclient: shutdown 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 1 -- 172.21.15.132:0/2847425598 >> v1:172.21.15.132:6789/0 conn(0x7f58880a98f0 legacy=0x7f58880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 1 -- 172.21.15.132:0/2847425598 shutdown_connections 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 1 -- 172.21.15.132:0/2847425598 >> 172.21.15.132:0/2847425598 conn(0x7f58880b7ea0 msgr2=0x7f58880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 1 -- 172.21.15.132:0/2847425598 shutdown_connections 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 1 -- 172.21.15.132:0/2847425598 wait complete. 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.108+0000 7f588eea8640 10 monclient: monmap: 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 1 librados: starting msgr at 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 1 librados: starting objecter 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 1 Processor -- start 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 1 -- start start 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 1 librados: setting wanted keys 2024-03-25T18:16:20.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 1 librados: calling monclient init 2024-03-25T18:16:20.112 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.109+0000 7f588eea8640 10 monclient: init 2024-03-25T18:16:20.112 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.112 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:20.112 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient(hunting): picked mon.noname-b con 0x7f58880a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient(hunting): picked mon.noname-a con 0x7f588813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient(hunting): picked mon.noname-c con 0x7f58881430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5888138c70 con 0x7f588813f710 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58881367d0 con 0x7f58880a98f0 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5888136960 con 0x7f58881430c0 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.110+0000 7f588eea8640 10 monclient(hunting): authenticate will time out at 7420.373118s 2024-03-25T18:16:20.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.120+0000 7f588cc44640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f58880a98f0 0x7f5888137610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:42682/0 (socket says 172.21.15.132:42682) 2024-03-25T18:16:20.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.120+0000 7f588cc44640 1 -- 172.21.15.132:0/196530437 learned_addr learned my addr 172.21.15.132:0/196530437 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:20.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.121+0000 7f58857fa640 10 client.?.objecter ms_handle_connect 0x7f58880a98f0 2024-03-25T18:16:20.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.121+0000 7f58857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.126+0000 7f58857fa640 10 client.?.objecter ms_handle_connect 0x7f588813f710 2024-03-25T18:16:20.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.126+0000 7f58857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.129+0000 7f58857fa640 10 client.?.objecter ms_handle_connect 0x7f58881430c0 2024-03-25T18:16:20.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.129+0000 7f58857fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.130+0000 7f588d445640 1 -- 172.21.15.132:0/196530437 >> v1:172.21.15.132:6790/0 conn(0x7f58881430c0 legacy=0x7f58881e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:16:20.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.130+0000 7f588d445640 1 -- 172.21.15.132:0/196530437 >> v1:172.21.15.132:6790/0 conn(0x7f58881430c0 legacy=0x7f58881e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:16:20.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.130+0000 7f588d445640 1 --1- 172.21.15.132:0/196530437 >> v1:172.21.15.132:6790/0 conn(0x7f58881430c0 0x7f58881e2cd0 :-1 s=OPENED pgs=16914 cs=1 l=1).handle_message read tag failed 2024-03-25T18:16:20.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.130+0000 7f588d445640 1 --1- 172.21.15.132:0/196530437 >> v1:172.21.15.132:6790/0 conn(0x7f58881430c0 0x7f58881e2cd0 :-1 s=OPENED pgs=16914 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:16:20.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.130+0000 7f58857fa640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.132:6790/0 2024-03-25T18:16:20.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.203+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356516953 0 0) 0x7f5888138c70 con 0x7f588813f710 2024-03-25T18:16:20.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.203+0000 7f58857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.203+0000 7f58857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.203+0000 7f58857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.203+0000 7f58857fa640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:16:20.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.203+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586c004eb0 con 0x7f588813f710 2024-03-25T18:16:20.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.212+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599071288 0 0) 0x7f58881367d0 con 0x7f58880a98f0 2024-03-25T18:16:20.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.212+0000 7f58857fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.212+0000 7f58857fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.212+0000 7f58857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.212+0000 7f58857fa640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:16:20.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.212+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5888138c70 con 0x7f58880a98f0 2024-03-25T18:16:20.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.213+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942714014 0 0) 0x7f586c004eb0 con 0x7f588813f710 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.214+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58881367d0 con 0x7f588813f710 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.214+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5874001930 con 0x7f588813f710 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.214+0000 7f58857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.214+0000 7f58857fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.214+0000 7f58857fa640 10 monclient(hunting): dump: 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:20.215 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:20.216 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:20.216 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:20.216 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.214+0000 7f58857fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.230+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2916958713 0 0) 0x7f5888138c70 con 0x7f58880a98f0 2024-03-25T18:16:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.230+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f586c004eb0 con 0x7f58880a98f0 2024-03-25T18:16:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.230+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f587c001b00 con 0x7f58880a98f0 2024-03-25T18:16:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.230+0000 7f58857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.230+0000 7f58857fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.230+0000 7f58857fa640 10 monclient(hunting): dump: 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:20.232 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 521266164 0 0) 0x7f58881367d0 con 0x7f588813f710 2024-03-25T18:16:20.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 >> v1:172.21.15.132:6790/0 conn(0x7f58881430c0 legacy=0x7f58881e2cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:16:20.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 >> v1:172.21.15.161:6789/0 conn(0x7f58880a98f0 legacy=0x7f5888137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 1 monclient: found mon.a 2024-03-25T18:16:20.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5888139630 con 0x7f588813f710 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f58857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 10 monclient: _renew_subs 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5888139880 con 0x7f588813f710 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 10 monclient: _renew_subs 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58881337f0 con 0x7f588813f710 2024-03-25T18:16:20.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.234+0000 7f588eea8640 1 librados: init done 2024-03-25T18:16:20.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.235+0000 7f588eea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.235+0000 7f588eea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.235+0000 7f588eea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:20.237 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.235+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5888133aa0 con 0x7f588813f710 2024-03-25T18:16:20.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58740035f0 con 0x7f588813f710 2024-03-25T18:16:20.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f58857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5874003980 con 0x7f588813f710 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f58857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f58857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f58857fa640 10 monclient: dump: 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:20.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.535+0000 7f5886ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:20.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.536+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f5874003ba0 con 0x7f588813f710 2024-03-25T18:16:20.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.537+0000 7f588cc44640 10 monclient: get_auth_request con 0x7f586c06de10 auth_method 0 2024-03-25T18:16:20.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.537+0000 7f58857fa640 10 client.4350.objecter ms_handle_connect 0x7f586c06de10 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.542+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f5874053550 con 0x7f588813f710 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.542+0000 7f58857fa640 10 client.4350.objecter ms_dispatch 0x7f5888000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.542+0000 7f58857fa640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.542+0000 7f58857fa640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.542+0000 7f58857fa640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.542+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f587404f060 con 0x7f588813f710 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.543+0000 7f58857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.543+0000 7f58857fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:20.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.795+0000 7f588eea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.795+0000 7f588eea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.795+0000 7f588eea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:20.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.795+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5888074cc0 con 0x7f588813f710 2024-03-25T18:16:20.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.832+0000 7f58857fa640 1 -- 172.21.15.132:0/196530437 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f5874050f90 con 0x7f588813f710 2024-03-25T18:16:20.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.832+0000 7f58857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:20.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.832+0000 7f58857fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:20.833 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:20.833 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 10 librados: watch_flush enter 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 10 librados: watch_flush exit 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 >> v1:172.21.15.161:6800/411950978 conn(0x7f586c06de10 legacy=0x7f586c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 10 monclient: shutdown 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.834+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 >> v1:172.21.15.132:6789/0 conn(0x7f588813f710 legacy=0x7f5888132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.835+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 shutdown_connections 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.835+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 >> 172.21.15.132:0/196530437 conn(0x7f58880b7ea0 msgr2=0x7f5888074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.835+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 shutdown_connections 2024-03-25T18:16:20.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.835+0000 7f588eea8640 1 -- 172.21.15.132:0/196530437 wait complete. 2024-03-25T18:16:20.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:20.835+0000 7f588eea8640 1 librados: shutdown 2024-03-25T18:16:20.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:27.655 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:27.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.768+0000 7f0067057640 10 monclient: get_monmap_and_config 2024-03-25T18:16:27.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.768+0000 7f0067057640 10 monclient: build_initial_monmap 2024-03-25T18:16:27.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.768+0000 7f0067057640 10 monclient: monmap: 2024-03-25T18:16:27.769 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:27.769 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:27.769433+0000 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:27.769433+0000 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:27.770 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.769+0000 7f0067057640 1 Processor -- start 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.769+0000 7f0067057640 1 -- start start 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.769+0000 7f0067057640 10 monclient: init 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient(hunting): picked mon.noname-c con 0x7f00600a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient(hunting): picked mon.noname-b con 0x7f006013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient(hunting): picked mon.noname-a con 0x7f00601430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0060142520 con 0x7f00601430c0 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00600b28a0 con 0x7f00600a98f0 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00600b2a30 con 0x7f006013f710 2024-03-25T18:16:27.771 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.772 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.770+0000 7f0067057640 10 monclient(hunting): authenticate will time out at 7428.032874s 2024-03-25T18:16:27.779 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.778+0000 7f00655f4640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f00601430c0 0x7f006014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37424/0 (socket says 172.21.15.132:37424) 2024-03-25T18:16:27.779 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.778+0000 7f00655f4640 1 -- 172.21.15.132:0/3420562553 learned_addr learned my addr 172.21.15.132:0/3420562553 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:27.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.793+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332861080 0 0) 0x7f0060142520 con 0x7f00601430c0 2024-03-25T18:16:27.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.793+0000 7f005f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.793+0000 7f005f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.793+0000 7f005f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.793+0000 7f005f7fe640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:16:27.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.793+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0050003520 con 0x7f00601430c0 2024-03-25T18:16:27.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.799+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073238926 0 0) 0x7f00600b2a30 con 0x7f006013f710 2024-03-25T18:16:27.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.799+0000 7f005f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.799+0000 7f005f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.799+0000 7f005f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.799+0000 7f005f7fe640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:16:27.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.799+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0060142520 con 0x7f006013f710 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730636677 0 0) 0x7f0050003520 con 0x7f00601430c0 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 >> v1:172.21.15.132:6790/0 conn(0x7f006013f710 legacy=0x7f00601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 >> v1:172.21.15.161:6789/0 conn(0x7f00600a98f0 legacy=0x7f00600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600b44c0 con 0x7f00601430c0 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0058001a10 con 0x7f00601430c0 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:27.804 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f005f7fe640 10 monclient: dump: 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f0067057640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:16:27.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.802+0000 7f0067057640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0058001bf0 con 0x7f00601430c0 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f005f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f005f7fe640 1 -- 172.21.15.132:0/3420562553 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0058002000 con 0x7f00601430c0 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f005f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f005f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f005f7fe640 10 monclient: dump: 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:27.814 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f0067057640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:27.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.813+0000 7f0067057640 4 set_mon_vals no callback set 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 10 monclient: shutdown 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 1 -- 172.21.15.132:0/3420562553 >> v1:172.21.15.132:6789/0 conn(0x7f00601430c0 legacy=0x7f006014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 1 -- 172.21.15.132:0/3420562553 shutdown_connections 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 1 -- 172.21.15.132:0/3420562553 >> 172.21.15.132:0/3420562553 conn(0x7f00600b7ea0 msgr2=0x7f00600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 1 -- 172.21.15.132:0/3420562553 shutdown_connections 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.814+0000 7f0067057640 1 -- 172.21.15.132:0/3420562553 wait complete. 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.815+0000 7f0067057640 10 monclient: build_initial_monmap 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.815+0000 7f0067057640 10 monclient: monmap: 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:27.816 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:27.817 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:27.817 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:27.817 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:27.817 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:27.817 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:27.817 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.815+0000 7f0067057640 1 librados: starting msgr at 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.815+0000 7f0067057640 1 librados: starting objecter 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 1 Processor -- start 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 1 -- start start 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 1 librados: setting wanted keys 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 1 librados: calling monclient init 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 10 monclient: init 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 10 monclient(hunting): picked mon.noname-c con 0x7f00600a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:27.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.816+0000 7f0067057640 10 monclient(hunting): picked mon.noname-a con 0x7f006013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:27.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.817+0000 7f0067057640 10 monclient(hunting): picked mon.noname-b con 0x7f00601430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:27.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.817+0000 7f0067057640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0060135070 con 0x7f006013f710 2024-03-25T18:16:27.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.817+0000 7f0067057640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00600b75c0 con 0x7f00601430c0 2024-03-25T18:16:27.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.817+0000 7f0067057640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00600b7750 con 0x7f00600a98f0 2024-03-25T18:16:27.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.817+0000 7f0067057640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.817+0000 7f0067057640 10 monclient(hunting): authenticate will time out at 7428.079769s 2024-03-25T18:16:27.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.822+0000 7f005ffff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f006013f710 0x7f00601d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37480/0 (socket says 172.21.15.132:37480) 2024-03-25T18:16:27.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.822+0000 7f005ffff640 1 -- 172.21.15.132:0/1318019363 learned_addr learned my addr 172.21.15.132:0/1318019363 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:27.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.825+0000 7f005d7fa640 10 client.?.objecter ms_handle_connect 0x7f006013f710 2024-03-25T18:16:27.826 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.825+0000 7f005d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.828+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1563701889 0 0) 0x7f0060135070 con 0x7f006013f710 2024-03-25T18:16:27.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.828+0000 7f005d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.828+0000 7f005d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.828+0000 7f005d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.828+0000 7f005d7fa640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:16:27.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.828+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0044004eb0 con 0x7f006013f710 2024-03-25T18:16:27.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.840+0000 7f005d7fa640 10 client.?.objecter ms_handle_connect 0x7f00600a98f0 2024-03-25T18:16:27.841 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.840+0000 7f005d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 10 client.?.objecter ms_handle_connect 0x7f00601430c0 2024-03-25T18:16:27.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:27.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 808304342 0 0) 0x7f0044004eb0 con 0x7f006013f710 2024-03-25T18:16:27.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0060135070 con 0x7f006013f710 2024-03-25T18:16:27.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f004c001a10 con 0x7f006013f710 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 10 monclient(hunting): dump: 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:27.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.850+0000 7f005d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:27.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.863+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737857407 0 0) 0x7f00600b75c0 con 0x7f00601430c0 2024-03-25T18:16:27.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.864+0000 7f005d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.864+0000 7f005d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.864+0000 7f005d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.864+0000 7f005d7fa640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:16:27.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.864+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0044004eb0 con 0x7f00601430c0 2024-03-25T18:16:27.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.865+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107013971 0 0) 0x7f00600b7750 con 0x7f00600a98f0 2024-03-25T18:16:27.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.865+0000 7f005d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:27.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.865+0000 7f005d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:27.866 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.865+0000 7f005d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:27.866 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.865+0000 7f005d7fa640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:16:27.866 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.865+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00600b75c0 con 0x7f00600a98f0 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3475810757 0 0) 0x7f0060135070 con 0x7f006013f710 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 >> v1:172.21.15.132:6790/0 conn(0x7f00600a98f0 legacy=0x7f00601d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 >> v1:172.21.15.161:6789/0 conn(0x7f00601430c0 legacy=0x7f00601e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 1 monclient: found mon.a 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.871+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0060135300 con 0x7f006013f710 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f005d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:27.873 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f005d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f005d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 10 monclient: _renew_subs 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0060135550 con 0x7f006013f710 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 10 monclient: _renew_subs 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f006013b620 con 0x7f006013f710 2024-03-25T18:16:27.874 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.872+0000 7f0067057640 1 librados: init done 2024-03-25T18:16:27.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.873+0000 7f0067057640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.873+0000 7f0067057640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:27.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.873+0000 7f0067057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:27.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:27.873+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00600627b0 con 0x7f006013f710 2024-03-25T18:16:28.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f004c001c90 con 0x7f006013f710 2024-03-25T18:16:28.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:28.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f004c003a60 con 0x7f006013f710 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 10 monclient: dump: 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:28.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005effd640 4 set_mon_vals no callback set 2024-03-25T18:16:28.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.176+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f004c003c80 con 0x7f006013f710 2024-03-25T18:16:28.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.177+0000 7f0064df3640 10 monclient: get_auth_request con 0x7f004406de10 auth_method 0 2024-03-25T18:16:28.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.177+0000 7f005d7fa640 10 client.4356.objecter ms_handle_connect 0x7f004406de10 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f004c0548b0 con 0x7f006013f710 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 10 client.4356.objecter ms_dispatch 0x7f0060000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f004c0503c0 con 0x7f006013f710 2024-03-25T18:16:28.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:28.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.183+0000 7f005d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:28.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.404+0000 7f0067057640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.404+0000 7f0067057640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.404+0000 7f0067057640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:16:28.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.404+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00600b7b10 con 0x7f006013f710 2024-03-25T18:16:28.426 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.425+0000 7f005d7fa640 1 -- 172.21.15.132:0/1318019363 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f004c0522f0 con 0x7f006013f710 2024-03-25T18:16:28.427 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.426+0000 7f005d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.427 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.426+0000 7f005d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:28.427 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:28.427 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:28.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 10 librados: watch_flush enter 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 10 librados: watch_flush exit 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 10 client.4356.objecter successfully canceled tick 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 >> v1:172.21.15.161:6800/411950978 conn(0x7f004406de10 legacy=0x7f00440702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 10 monclient: shutdown 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 >> v1:172.21.15.132:6789/0 conn(0x7f006013f710 legacy=0x7f00601d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 shutdown_connections 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 >> 172.21.15.132:0/1318019363 conn(0x7f00600b7ea0 msgr2=0x7f0060074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 shutdown_connections 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 -- 172.21.15.132:0/1318019363 wait complete. 2024-03-25T18:16:28.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:28.428+0000 7f0067057640 1 librados: shutdown 2024-03-25T18:16:28.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:34.443 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:19.707 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.454 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:19.718 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.503 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:19.767 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.505 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:19.770 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.507 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:19.749 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.519 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:19.761 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.520 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:19.784 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.548 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:19.812 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:34.551 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:19.793 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:35.245 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:35.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.359+0000 7fd7a099d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:35.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.359+0000 7fd7a099d640 10 monclient: build_initial_monmap 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.359+0000 7fd7a099d640 10 monclient: monmap: 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:35.360892+0000 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:35.360892+0000 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:35.361 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:35.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.360+0000 7fd7a099d640 1 Processor -- start 2024-03-25T18:16:35.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.360+0000 7fd7a099d640 1 -- start start 2024-03-25T18:16:35.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.360+0000 7fd7a099d640 10 monclient: init 2024-03-25T18:16:35.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient(hunting): picked mon.noname-b con 0x7fd79c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient(hunting): picked mon.noname-a con 0x7fd79c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c1430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c142520 con 0x7fd79c13f710 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c0b28a0 con 0x7fd79c1430c0 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c0b2a30 con 0x7fd79c0a98f0 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:35.363 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.361+0000 7fd7a099d640 10 monclient(hunting): authenticate will time out at 7435.624557s 2024-03-25T18:16:35.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.371+0000 7fd793fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd79c13f710 0x7fd79c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:51020/0 (socket says 172.21.15.132:51020) 2024-03-25T18:16:35.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.371+0000 7fd793fff640 1 -- 172.21.15.132:0/2574762706 learned_addr learned my addr 172.21.15.132:0/2574762706 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:35.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.390+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150350126 0 0) 0x7fd79c0b2a30 con 0x7fd79c0a98f0 2024-03-25T18:16:35.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.390+0000 7fd799d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.390+0000 7fd799d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.390+0000 7fd799d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.390+0000 7fd799d9b640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:16:35.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.390+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003520 con 0x7fd79c0a98f0 2024-03-25T18:16:35.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.397+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993193253 0 0) 0x7fd79c142520 con 0x7fd79c13f710 2024-03-25T18:16:35.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.397+0000 7fd799d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.397+0000 7fd799d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.397+0000 7fd799d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.397+0000 7fd799d9b640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:16:35.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.397+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd79c0b2a30 con 0x7fd79c13f710 2024-03-25T18:16:35.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.399+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191581282 0 0) 0x7fd79c0b28a0 con 0x7fd79c1430c0 2024-03-25T18:16:35.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.399+0000 7fd799d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.399+0000 7fd799d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.399+0000 7fd799d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.399+0000 7fd799d9b640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:16:35.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.399+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd79c142520 con 0x7fd79c1430c0 2024-03-25T18:16:35.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2914498291 0 0) 0x7fd79c142520 con 0x7fd79c1430c0 2024-03-25T18:16:35.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 >> v1:172.21.15.132:6790/0 conn(0x7fd79c0a98f0 legacy=0x7fd79c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 >> v1:172.21.15.132:6789/0 conn(0x7fd79c13f710 legacy=0x7fd79c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.418 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 1 monclient: found mon.noname-c 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c0b44c0 con 0x7fd79c1430c0 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd794001930 con 0x7fd79c1430c0 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd799d9b640 10 monclient: dump: 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.419 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd7a099d640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:16:35.420 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.417+0000 7fd7a099d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd794001a90 con 0x7fd79c1430c0 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd799d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd799d9b640 1 -- 172.21.15.132:0/2574762706 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd794001ee0 con 0x7fd79c1430c0 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd799d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd799d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd799d9b640 10 monclient: dump: 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:35.436 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd7a099d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:35.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.435+0000 7fd7a099d640 4 set_mon_vals no callback set 2024-03-25T18:16:35.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 10 monclient: shutdown 2024-03-25T18:16:35.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 1 -- 172.21.15.132:0/2574762706 >> v1:172.21.15.161:6789/0 conn(0x7fd79c1430c0 legacy=0x7fd79c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 1 -- 172.21.15.132:0/2574762706 shutdown_connections 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 1 -- 172.21.15.132:0/2574762706 >> 172.21.15.132:0/2574762706 conn(0x7fd79c0b7ea0 msgr2=0x7fd79c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 1 -- 172.21.15.132:0/2574762706 shutdown_connections 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.436+0000 7fd7a099d640 1 -- 172.21.15.132:0/2574762706 wait complete. 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 10 monclient: build_initial_monmap 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 10 monclient: monmap: 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:35.440 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 1 librados: starting msgr at 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 1 librados: starting objecter 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 1 Processor -- start 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 1 -- start start 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 1 librados: setting wanted keys 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 1 librados: calling monclient init 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.437+0000 7fd7a099d640 10 monclient: init 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:35.441 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient(hunting): picked mon.noname-a con 0x7fd79c0a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient(hunting): picked mon.noname-c con 0x7fd79c13f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient(hunting): picked mon.noname-b con 0x7fd79c1430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c0abe50 con 0x7fd79c0a98f0 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c138f20 con 0x7fd79c1430c0 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd79c1d51f0 con 0x7fd79c13f710 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.438+0000 7fd7a099d640 10 monclient(hunting): authenticate will time out at 7435.701392s 2024-03-25T18:16:35.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.440+0000 7fd79a59c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd79c0a98f0 0x7fd79c138810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:51082/0 (socket says 172.21.15.132:51082) 2024-03-25T18:16:35.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.440+0000 7fd79a59c640 1 -- 172.21.15.132:0/2725585280 learned_addr learned my addr 172.21.15.132:0/2725585280 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:35.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.441+0000 7fd792ffd640 10 client.?.objecter ms_handle_connect 0x7fd79c0a98f0 2024-03-25T18:16:35.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.441+0000 7fd792ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.447+0000 7fd792ffd640 10 client.?.objecter ms_handle_connect 0x7fd79c1430c0 2024-03-25T18:16:35.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.447+0000 7fd792ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.507+0000 7fd792ffd640 10 client.?.objecter ms_handle_connect 0x7fd79c13f710 2024-03-25T18:16:35.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.507+0000 7fd792ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.519+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806574896 0 0) 0x7fd79c1d51f0 con 0x7fd79c13f710 2024-03-25T18:16:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.519+0000 7fd792ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.519+0000 7fd792ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.519+0000 7fd792ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.519+0000 7fd792ffd640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:16:35.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.519+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd77c004eb0 con 0x7fd79c13f710 2024-03-25T18:16:35.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2352064057 0 0) 0x7fd77c004eb0 con 0x7fd79c13f710 2024-03-25T18:16:35.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd79c1d51f0 con 0x7fd79c13f710 2024-03-25T18:16:35.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd784001930 con 0x7fd79c13f710 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:35.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.542+0000 7fd792ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2186272386 0 0) 0x7fd79c1d51f0 con 0x7fd79c13f710 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 >> v1:172.21.15.161:6789/0 conn(0x7fd79c1430c0 legacy=0x7fd79c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 >> v1:172.21.15.132:6789/0 conn(0x7fd79c0a98f0 legacy=0x7fd79c138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 1 monclient: found mon.c 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:35.559 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79c1d7730 con 0x7fd79c13f710 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd792ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 5 monclient: authenticate success, global_id 4352 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 10 monclient: _renew_subs 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd79c1d8360 con 0x7fd79c13f710 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 10 monclient: _renew_subs 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd79c137ce0 con 0x7fd79c13f710 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.557+0000 7fd7a099d640 1 librados: init done 2024-03-25T18:16:35.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.558+0000 7fd7a099d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.558+0000 7fd7a099d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.558+0000 7fd7a099d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:35.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.558+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd79c1d7590 con 0x7fd79c13f710 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7840047e0 con 0x7fd79c13f710 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd792ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd784004a90 con 0x7fd79c13f710 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd792ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd792ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd792ffd640 10 monclient: dump: 2024-03-25T18:16:35.810 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:35.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.809+0000 7fd79959a640 4 set_mon_vals no callback set 2024-03-25T18:16:35.812 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.810+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fd784004cb0 con 0x7fd79c13f710 2024-03-25T18:16:35.812 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.811+0000 7fd79a59c640 10 monclient: get_auth_request con 0x7fd77c06dd00 auth_method 0 2024-03-25T18:16:35.812 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.811+0000 7fd792ffd640 10 client.4352.objecter ms_handle_connect 0x7fd77c06dd00 2024-03-25T18:16:35.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fd7840546b0 con 0x7fd79c13f710 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 10 client.4352.objecter ms_dispatch 0x7fd79c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7840501c0 con 0x7fd79c13f710 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:35.828+0000 7fd792ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:36.096 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.095+0000 7fd7a099d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.096 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.095+0000 7fd7a099d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.096 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.095+0000 7fd7a099d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:36.096 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.095+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd79c1d7970 con 0x7fd79c13f710 2024-03-25T18:16:36.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.127+0000 7fd792ffd640 1 -- 172.21.15.132:0/2725585280 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fd79c1d7970 con 0x7fd79c13f710 2024-03-25T18:16:36.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.127+0000 7fd792ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.127+0000 7fd792ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:36.128 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:36.128 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.129+0000 7fd7a099d640 10 librados: watch_flush enter 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.129+0000 7fd7a099d640 10 librados: watch_flush exit 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.129+0000 7fd7a099d640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.129+0000 7fd7a099d640 10 client.4352.objecter successfully canceled tick 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 >> v1:172.21.15.161:6800/411950978 conn(0x7fd77c06dd00 legacy=0x7fd77c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 10 monclient: shutdown 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 >> v1:172.21.15.132:6790/0 conn(0x7fd79c13f710 legacy=0x7fd79c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 shutdown_connections 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 >> 172.21.15.132:0/2725585280 conn(0x7fd79c0b7ea0 msgr2=0x7fd79c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:36.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 shutdown_connections 2024-03-25T18:16:36.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 -- 172.21.15.132:0/2725585280 wait complete. 2024-03-25T18:16:36.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:36.130+0000 7fd7a099d640 1 librados: shutdown 2024-03-25T18:16:36.144 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:42.951 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:43.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.066+0000 7f61514c9640 10 monclient: get_monmap_and_config 2024-03-25T18:16:43.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.066+0000 7f61514c9640 10 monclient: build_initial_monmap 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.066+0000 7f61514c9640 10 monclient: monmap: 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:43.067678+0000 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:43.067678+0000 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:43.068 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:43.069 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 1 Processor -- start 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 1 -- start start 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 10 monclient: init 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 10 monclient(hunting): picked mon.noname-c con 0x7f614c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:43.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 10 monclient(hunting): picked mon.noname-b con 0x7f614c13f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:43.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 10 monclient(hunting): picked mon.noname-a con 0x7f614c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:43.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c142520 con 0x7f614c1430c0 2024-03-25T18:16:43.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c0b28a0 con 0x7f614c0a98f0 2024-03-25T18:16:43.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.067+0000 7f61514c9640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c0b2a30 con 0x7f614c13f710 2024-03-25T18:16:43.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.068+0000 7f61514c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:43.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.068+0000 7f61514c9640 10 monclient(hunting): authenticate will time out at 7443.330724s 2024-03-25T18:16:43.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.073+0000 7f614b7fe640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f614c1430c0 0x7f614c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:49366/0 (socket says 172.21.15.132:49366) 2024-03-25T18:16:43.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.073+0000 7f614b7fe640 1 -- 172.21.15.132:0/2771886097 learned_addr learned my addr 172.21.15.132:0/2771886097 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:43.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.084+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251982368 0 0) 0x7f614c0b28a0 con 0x7f614c0a98f0 2024-03-25T18:16:43.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.084+0000 7f6149ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.084+0000 7f6149ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.084+0000 7f6149ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.084+0000 7f6149ffb640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:16:43.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.084+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6134003520 con 0x7f614c0a98f0 2024-03-25T18:16:43.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674238159 0 0) 0x7f6134003520 con 0x7f614c0a98f0 2024-03-25T18:16:43.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 >> v1:172.21.15.132:6790/0 conn(0x7f614c13f710 legacy=0x7f614c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 >> v1:172.21.15.132:6789/0 conn(0x7f614c1430c0 legacy=0x7f614c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:43.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 1 monclient: found mon.noname-c 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c0b44c0 con 0x7f614c0a98f0 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f613c001690 con 0x7f614c0a98f0 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f61514c9640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f61514c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f6149ffb640 10 monclient: dump: 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:43.094 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:43.095 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:43.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.091+0000 7f61514c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f613c0017f0 con 0x7f614c0a98f0 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f6149ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f6149ffb640 1 -- 172.21.15.132:0/2771886097 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f613c001c40 con 0x7f614c0a98f0 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f6149ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f6149ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f6149ffb640 10 monclient: dump: 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:43.102 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f61514c9640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:43.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f61514c9640 4 set_mon_vals no callback set 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f61514c9640 10 monclient: shutdown 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f61514c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.101+0000 7f61514c9640 1 -- 172.21.15.132:0/2771886097 >> v1:172.21.15.161:6789/0 conn(0x7f614c0a98f0 legacy=0x7f614c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.102+0000 7f61514c9640 1 -- 172.21.15.132:0/2771886097 shutdown_connections 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.102+0000 7f61514c9640 1 -- 172.21.15.132:0/2771886097 >> 172.21.15.132:0/2771886097 conn(0x7f614c0b7ea0 msgr2=0x7f614c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.102+0000 7f61514c9640 1 -- 172.21.15.132:0/2771886097 shutdown_connections 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.102+0000 7f61514c9640 1 -- 172.21.15.132:0/2771886097 wait complete. 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.102+0000 7f61514c9640 10 monclient: build_initial_monmap 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 10 monclient: monmap: 2024-03-25T18:16:43.106 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 1 librados: starting msgr at 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 1 librados: starting objecter 2024-03-25T18:16:43.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 1 Processor -- start 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 1 -- start start 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 1 librados: setting wanted keys 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 1 librados: calling monclient init 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.103+0000 7f61514c9640 10 monclient: init 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient(hunting): picked mon.noname-c con 0x7f614c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient(hunting): picked mon.noname-b con 0x7f614c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient(hunting): picked mon.noname-a con 0x7f614c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c138b20 con 0x7f614c1430c0 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c137a80 con 0x7f614c13f710 2024-03-25T18:16:43.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614c139880 con 0x7f614c0a98f0 2024-03-25T18:16:43.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:43.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.104+0000 7f61514c9640 10 monclient(hunting): authenticate will time out at 7443.367100s 2024-03-25T18:16:43.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.114+0000 7f614a7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f614c13f710 0x7f614c139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:36462/0 (socket says 172.21.15.132:36462) 2024-03-25T18:16:43.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.114+0000 7f614a7fc640 1 -- 172.21.15.132:0/310960277 learned_addr learned my addr 172.21.15.132:0/310960277 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:43.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.115+0000 7f61337fe640 10 client.?.objecter ms_handle_connect 0x7f614c13f710 2024-03-25T18:16:43.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.115+0000 7f61337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.121+0000 7f61337fe640 10 client.?.objecter ms_handle_connect 0x7f614c0a98f0 2024-03-25T18:16:43.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.121+0000 7f61337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.122+0000 7f61337fe640 10 client.?.objecter ms_handle_connect 0x7f614c1430c0 2024-03-25T18:16:43.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.122+0000 7f61337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.130+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645925771 0 0) 0x7f614c137a80 con 0x7f614c13f710 2024-03-25T18:16:43.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.130+0000 7f61337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.130+0000 7f61337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.130+0000 7f61337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.130+0000 7f61337fe640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:16:43.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.130+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6138004eb0 con 0x7f614c13f710 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.146+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777944025 0 0) 0x7f614c138b20 con 0x7f614c1430c0 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.146+0000 7f61337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.146+0000 7f61337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.146+0000 7f61337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.146+0000 7f61337fe640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:16:43.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.146+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f614c137a80 con 0x7f614c1430c0 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.147+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475465224 0 0) 0x7f614c139880 con 0x7f614c0a98f0 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.147+0000 7f61337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.147+0000 7f61337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.147+0000 7f61337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.147+0000 7f61337fe640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:16:43.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.147+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f614c138b20 con 0x7f614c0a98f0 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 965188339 0 0) 0x7f6138004eb0 con 0x7f614c13f710 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f614c139880 con 0x7f614c13f710 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6140001a30 con 0x7f614c13f710 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:43.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 10 monclient(hunting): dump: 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:43.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.157+0000 7f61337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1571956600 0 0) 0x7f614c139880 con 0x7f614c13f710 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 >> v1:172.21.15.132:6790/0 conn(0x7f614c0a98f0 legacy=0x7f614c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 >> v1:172.21.15.132:6789/0 conn(0x7f614c1430c0 legacy=0x7f614c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 1 monclient: found mon.b 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c139a10 con 0x7f614c13f710 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:43.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 10 monclient: _renew_subs 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c1dab00 con 0x7f614c13f710 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 10 monclient: _renew_subs 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c13b4f0 con 0x7f614c13f710 2024-03-25T18:16:43.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.170+0000 7f61514c9640 1 librados: init done 2024-03-25T18:16:43.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.171+0000 7f61514c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.171+0000 7f61514c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.171+0000 7f61514c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:43.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.171+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c0627b0 con 0x7f614c13f710 2024-03-25T18:16:43.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6140001c70 con 0x7f614c13f710 2024-03-25T18:16:43.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6140003a60 con 0x7f614c13f710 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61337fe640 10 monclient: dump: 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:43.481 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:43.482 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:43.482 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:43.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.479+0000 7f61497fa640 4 set_mon_vals no callback set 2024-03-25T18:16:43.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.480+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f6140003c40 con 0x7f614c13f710 2024-03-25T18:16:43.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.481+0000 7f614affd640 10 monclient: get_auth_request con 0x7f613806ddf0 auth_method 0 2024-03-25T18:16:43.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.481+0000 7f61337fe640 10 client.4366.objecter ms_handle_connect 0x7f613806ddf0 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f61400548b0 con 0x7f614c13f710 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 10 client.4366.objecter ms_dispatch 0x7f614c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f61400503c0 con 0x7f614c13f710 2024-03-25T18:16:43.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.487 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.485+0000 7f61337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:43.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.696+0000 7f61514c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.696+0000 7f61514c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.696+0000 7f61514c9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:43.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.696+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f614c0b3ed0 con 0x7f614c13f710 2024-03-25T18:16:43.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.722+0000 7f61337fe640 1 -- 172.21.15.132:0/310960277 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f61400522f0 con 0x7f614c13f710 2024-03-25T18:16:43.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.723+0000 7f61337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:43.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.723+0000 7f61337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:43.724 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:43.724 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:43.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 10 librados: watch_flush enter 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 10 librados: watch_flush exit 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 10 client.4366.objecter successfully canceled tick 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 >> v1:172.21.15.161:6800/411950978 conn(0x7f613806ddf0 legacy=0x7f61380702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 10 monclient: shutdown 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 >> v1:172.21.15.161:6789/0 conn(0x7f614c13f710 legacy=0x7f614c139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 shutdown_connections 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 >> 172.21.15.132:0/310960277 conn(0x7f614c0b7ea0 msgr2=0x7f614c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 shutdown_connections 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 -- 172.21.15.132:0/310960277 wait complete. 2024-03-25T18:16:43.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:43.725+0000 7f61514c9640 1 librados: shutdown 2024-03-25T18:16:43.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:50.544 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.659+0000 7f1bcb3b6640 10 monclient: get_monmap_and_config 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.659+0000 7f1bcb3b6640 10 monclient: build_initial_monmap 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.659+0000 7f1bcb3b6640 10 monclient: monmap: 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:50.660203+0000 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:50.660203+0000 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:50.660 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.659+0000 7f1bcb3b6640 1 Processor -- start 2024-03-25T18:16:50.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 1 -- start start 2024-03-25T18:16:50.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient: init 2024-03-25T18:16:50.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:50.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc40a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc413f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc41430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc4142520 con 0x7f1bc40a98f0 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40b28a0 con 0x7f1bc41430c0 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40b2a30 con 0x7f1bc413f710 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:50.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.660+0000 7f1bcb3b6640 10 monclient(hunting): authenticate will time out at 7450.923695s 2024-03-25T18:16:50.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.675+0000 7f1bc9152640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f1bc40a98f0 0x7f1bc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54250/0 (socket says 172.21.15.132:54250) 2024-03-25T18:16:50.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.675+0000 7f1bc9152640 1 -- 172.21.15.132:0/3523530814 learned_addr learned my addr 172.21.15.132:0/3523530814 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:50.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.687+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717809216 0 0) 0x7f1bc40b2a30 con 0x7f1bc413f710 2024-03-25T18:16:50.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.687+0000 7f1bbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.687+0000 7f1bbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.687+0000 7f1bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.687+0000 7f1bbbfff640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:16:50.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.687+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0003520 con 0x7f1bc413f710 2024-03-25T18:16:50.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.695+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140183475 0 0) 0x7f1bc4142520 con 0x7f1bc40a98f0 2024-03-25T18:16:50.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.695+0000 7f1bbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.695+0000 7f1bbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.695+0000 7f1bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.695+0000 7f1bbbfff640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:16:50.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.695+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc40b2a30 con 0x7f1bc40a98f0 2024-03-25T18:16:50.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.696+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476264889 0 0) 0x7f1bc40b28a0 con 0x7f1bc41430c0 2024-03-25T18:16:50.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.696+0000 7f1bbbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.696+0000 7f1bbbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.696+0000 7f1bbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.696+0000 7f1bbbfff640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:16:50.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.696+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc4142520 con 0x7f1bc41430c0 2024-03-25T18:16:50.711 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3988802421 0 0) 0x7f1bb0003520 con 0x7f1bc413f710 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 >> v1:172.21.15.161:6789/0 conn(0x7f1bc41430c0 legacy=0x7f1bc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 >> v1:172.21.15.132:6789/0 conn(0x7f1bc40a98f0 legacy=0x7f1bc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 1 monclient: found mon.noname-b 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40b44c0 con 0x7f1bc413f710 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1bb4001690 con 0x7f1bc413f710 2024-03-25T18:16:50.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bcb3b6640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bcb3b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bbbfff640 10 monclient: dump: 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:50.713 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.710+0000 7f1bcb3b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:50.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bb40017f0 con 0x7f1bc413f710 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bbbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bbbfff640 1 -- 172.21.15.132:0/3523530814 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1bb4001c00 con 0x7f1bc413f710 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 4 set_mon_vals no callback set 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bbbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bbbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bbbfff640 10 monclient: dump: 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:50.721 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:50.722 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:50.722 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 10 monclient: shutdown 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3523530814 >> v1:172.21.15.132:6790/0 conn(0x7f1bc413f710 legacy=0x7f1bc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3523530814 shutdown_connections 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.720+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3523530814 >> 172.21.15.132:0/3523530814 conn(0x7f1bc40b7ea0 msgr2=0x7f1bc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.721+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3523530814 shutdown_connections 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.721+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3523530814 wait complete. 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.721+0000 7f1bcb3b6640 10 monclient: build_initial_monmap 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.721+0000 7f1bcb3b6640 10 monclient: monmap: 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:50.724 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 librados: starting msgr at 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 librados: starting objecter 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 Processor -- start 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 -- start start 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 librados: setting wanted keys 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 librados: calling monclient init 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient: init 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:50.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc40a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc41430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40b1870 con 0x7f1bc413f710 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40ac910 con 0x7f1bc40a98f0 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40abb00 con 0x7f1bc41430c0 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:50.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.722+0000 7f1bcb3b6640 10 monclient(hunting): authenticate will time out at 7450.985636s 2024-03-25T18:16:50.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.742+0000 7f1bc9152640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1bc40a98f0 0x7f1bc4138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41410/0 (socket says 172.21.15.132:41410) 2024-03-25T18:16:50.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.742+0000 7f1bc9152640 1 -- 172.21.15.132:0/3671728310 learned_addr learned my addr 172.21.15.132:0/3671728310 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:50.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.744+0000 7f1bb9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bc40a98f0 2024-03-25T18:16:50.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.744+0000 7f1bb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.745+0000 7f1bb9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bc41430c0 2024-03-25T18:16:50.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.745+0000 7f1bb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.747+0000 7f1bb9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bc413f710 2024-03-25T18:16:50.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.747+0000 7f1bb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:50.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.756+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636264772 0 0) 0x7f1bc40b44c0 con 0x7f1bc413f710 2024-03-25T18:16:50.757 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.756+0000 7f1bb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.756+0000 7f1bb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.756+0000 7f1bb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.756+0000 7f1bb9ffb640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:16:50.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.757+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba8004eb0 con 0x7f1bc413f710 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155762852 0 0) 0x7f1bc40ac910 con 0x7f1bc40a98f0 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc40b44c0 con 0x7f1bc40a98f0 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404766547 0 0) 0x7f1bc40abb00 con 0x7f1bc41430c0 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:50.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:16:50.762 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.760+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc40ac910 con 0x7f1bc41430c0 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4002576335 0 0) 0x7f1bc40b44c0 con 0x7f1bc40a98f0 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc40abb00 con 0x7f1bc40a98f0 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1bc0001970 con 0x7f1bc40a98f0 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:50.777 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:50.778 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.778 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.776+0000 7f1bb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.782+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 339752458 0 0) 0x7f1ba8004eb0 con 0x7f1bc413f710 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.782+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc40b44c0 con 0x7f1bc413f710 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.782+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1bb4001960 con 0x7f1bc413f710 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.782+0000 7f1bb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.782+0000 7f1bb9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.782+0000 7f1bb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.783 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.783+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4069924908 0 0) 0x7f1bc40ac910 con 0x7f1bc41430c0 2024-03-25T18:16:50.784 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.783+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ba8004eb0 con 0x7f1bc41430c0 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.783+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1bbc001b50 con 0x7f1bc41430c0 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.783+0000 7f1bb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.783+0000 7f1bb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.783+0000 7f1bb9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:50.785 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 87670725 0 0) 0x7f1bc40abb00 con 0x7f1bc40a98f0 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 >> v1:172.21.15.132:6790/0 conn(0x7f1bc41430c0 legacy=0x7f1bc41e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 >> v1:172.21.15.132:6789/0 conn(0x7f1bc413f710 legacy=0x7f1bc40ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 1 monclient: found mon.b 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:50.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40abde0 con 0x7f1bc40a98f0 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bcb3b6640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bcb3b6640 10 monclient: _renew_subs 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bcb3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.795+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc4138f40 con 0x7f1bc40a98f0 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.796+0000 7f1bcb3b6640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.796+0000 7f1bcb3b6640 10 monclient: _renew_subs 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.796+0000 7f1bcb3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.796+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc41363f0 con 0x7f1bc40a98f0 2024-03-25T18:16:50.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.796+0000 7f1bcb3b6640 1 librados: init done 2024-03-25T18:16:50.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.797+0000 7f1bcb3b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.797+0000 7f1bcb3b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:50.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.797+0000 7f1bcb3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:50.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:50.797+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc40627b0 con 0x7f1bc40a98f0 2024-03-25T18:16:51.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bc00047e0 con 0x7f1bc40a98f0 2024-03-25T18:16:51.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1bc0004af0 con 0x7f1bc40a98f0 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bb9ffb640 10 monclient: dump: 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:51.071 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:51.072 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:51.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.069+0000 7f1bbb7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:51.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.070+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f1bc001aac0 con 0x7f1bc40a98f0 2024-03-25T18:16:51.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.071+0000 7f1bc8951640 10 monclient: get_auth_request con 0x7f1ba806e090 auth_method 0 2024-03-25T18:16:51.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.071+0000 7f1bb9ffb640 10 client.4372.objecter ms_handle_connect 0x7f1ba806e090 2024-03-25T18:16:51.077 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.076+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f1bc00547e0 con 0x7f1bc40a98f0 2024-03-25T18:16:51.077 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.076+0000 7f1bb9ffb640 10 client.4372.objecter ms_dispatch 0x7f1bc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:51.077 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.076+0000 7f1bb9ffb640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:51.077 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.076+0000 7f1bb9ffb640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:51.078 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.076+0000 7f1bb9ffb640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:16:51.078 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.077+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bc0050120 con 0x7f1bc40a98f0 2024-03-25T18:16:51.078 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.077+0000 7f1bb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.078 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.077+0000 7f1bb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:51.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.313+0000 7f1bcb3b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.313+0000 7f1bcb3b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.313+0000 7f1bcb3b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:16:51.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.313+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc41434a0 con 0x7f1bc40a98f0 2024-03-25T18:16:51.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.332+0000 7f1bb9ffb640 1 -- 172.21.15.132:0/3671728310 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f1bc0052220 con 0x7f1bc40a98f0 2024-03-25T18:16:51.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.332+0000 7f1bb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.332+0000 7f1bb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:51.333 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:51.333 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:51.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 10 librados: watch_flush enter 2024-03-25T18:16:51.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 10 librados: watch_flush exit 2024-03-25T18:16:51.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:16:51.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:16:51.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 >> v1:172.21.15.161:6800/411950978 conn(0x7f1ba806e090 legacy=0x7f1ba8070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 10 monclient: shutdown 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 >> v1:172.21.15.161:6789/0 conn(0x7f1bc40a98f0 legacy=0x7f1bc4138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 shutdown_connections 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 >> 172.21.15.132:0/3671728310 conn(0x7f1bc40b7ea0 msgr2=0x7f1bc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 shutdown_connections 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 -- 172.21.15.132:0/3671728310 wait complete. 2024-03-25T18:16:51.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:51.334+0000 7f1bcb3b6640 1 librados: shutdown 2024-03-25T18:16:51.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:54.513 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:39.777 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.519 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:39.784 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.558 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:39.822 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.583 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:39.824 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.587 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:39.829 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.606 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:39.870 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.610 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:39.875 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.615 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:39.857 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:54.650 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:39.914 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:58.153 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.267+0000 7f4287159640 10 monclient: get_monmap_and_config 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.267+0000 7f4287159640 10 monclient: build_initial_monmap 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.267+0000 7f4287159640 10 monclient: monmap: 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:16:58.268975+0000 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:16:58.268975+0000 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:16:58.269 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:58.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 1 Processor -- start 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 1 -- start start 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 10 monclient: init 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 10 monclient(hunting): picked mon.noname-c con 0x7f42800a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 10 monclient(hunting): picked mon.noname-a con 0x7f428013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 10 monclient(hunting): picked mon.noname-b con 0x7f42801430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.269+0000 7f4287159640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4280142520 con 0x7f428013f710 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.270+0000 7f4287159640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42800b28a0 con 0x7f42800a98f0 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.270+0000 7f4287159640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42800b2a30 con 0x7f42801430c0 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.270+0000 7f4287159640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:58.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.270+0000 7f4287159640 10 monclient(hunting): authenticate will time out at 7458.532740s 2024-03-25T18:16:58.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.279+0000 7f4284ef5640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f42800a98f0 0x7f42800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:39836/0 (socket says 172.21.15.132:39836) 2024-03-25T18:16:58.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.279+0000 7f4284ef5640 1 -- 172.21.15.132:0/1383404542 learned_addr learned my addr 172.21.15.132:0/1383404542 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:58.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.294+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144514097 0 0) 0x7f4280142520 con 0x7f428013f710 2024-03-25T18:16:58.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.294+0000 7f427f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.294+0000 7f427f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.294+0000 7f427f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.294+0000 7f427f7fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:16:58.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.294+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4270003520 con 0x7f428013f710 2024-03-25T18:16:58.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.302+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783839559 0 0) 0x7f42800b28a0 con 0x7f42800a98f0 2024-03-25T18:16:58.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.302+0000 7f427f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.302+0000 7f427f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.302+0000 7f427f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.302+0000 7f427f7fe640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:16:58.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.302+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4280142520 con 0x7f42800a98f0 2024-03-25T18:16:58.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.307+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104866465 0 0) 0x7f42800b2a30 con 0x7f42801430c0 2024-03-25T18:16:58.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.307+0000 7f427f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.307+0000 7f427f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.307+0000 7f427f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.307+0000 7f427f7fe640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:16:58.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.307+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42800b28a0 con 0x7f42801430c0 2024-03-25T18:16:58.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241578797 0 0) 0x7f4280142520 con 0x7f42800a98f0 2024-03-25T18:16:58.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 >> v1:172.21.15.132:6790/0 conn(0x7f42801430c0 legacy=0x7f428014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 >> v1:172.21.15.132:6789/0 conn(0x7f428013f710 legacy=0x7f42801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800b44c0 con 0x7f42800a98f0 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4274001670 con 0x7f42800a98f0 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f427f7fe640 10 monclient: dump: 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.315 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f4287159640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:16:58.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.313+0000 7f4287159640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42740017d0 con 0x7f42800a98f0 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f427f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f427f7fe640 1 -- 172.21.15.132:0/1383404542 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4274001be0 con 0x7f42800a98f0 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f4287159640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f4287159640 4 set_mon_vals no callback set 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f427f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f427f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.335+0000 7f427f7fe640 10 monclient: dump: 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 10 monclient: shutdown 2024-03-25T18:16:58.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:58.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 1 -- 172.21.15.132:0/1383404542 >> v1:172.21.15.161:6789/0 conn(0x7f42800a98f0 legacy=0x7f42800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 1 -- 172.21.15.132:0/1383404542 shutdown_connections 2024-03-25T18:16:58.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 1 -- 172.21.15.132:0/1383404542 >> 172.21.15.132:0/1383404542 conn(0x7f42800b7ea0 msgr2=0x7f42800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:58.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 1 -- 172.21.15.132:0/1383404542 shutdown_connections 2024-03-25T18:16:58.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.336+0000 7f4287159640 1 -- 172.21.15.132:0/1383404542 wait complete. 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.337+0000 7f4287159640 10 monclient: build_initial_monmap 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.337+0000 7f4287159640 10 monclient: monmap: 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:58.340 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.337+0000 7f4287159640 1 librados: starting msgr at 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.337+0000 7f4287159640 1 librados: starting objecter 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.337+0000 7f4287159640 1 Processor -- start 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 1 -- start start 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 1 librados: setting wanted keys 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 1 librados: calling monclient init 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient: init 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient(hunting): picked mon.noname-a con 0x7f42800a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient(hunting): picked mon.noname-b con 0x7f428013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:16:58.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient(hunting): picked mon.noname-c con 0x7f42801430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:16:58.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4280139330 con 0x7f42800a98f0 2024-03-25T18:16:58.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42800abe70 con 0x7f428013f710 2024-03-25T18:16:58.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42800ac150 con 0x7f42801430c0 2024-03-25T18:16:58.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:58.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.338+0000 7f4287159640 10 monclient(hunting): authenticate will time out at 7458.601551s 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.349+0000 7f42856f6640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f42801430c0 0x7f42801e2510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:40824/0 (socket says 172.21.15.132:40824) 2024-03-25T18:16:58.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.349+0000 7f42856f6640 1 -- 172.21.15.132:0/4093008850 learned_addr learned my addr 172.21.15.132:0/4093008850 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.351+0000 7f427d7fa640 10 client.?.objecter ms_handle_connect 0x7f42801430c0 2024-03-25T18:16:58.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.351+0000 7f427d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.359+0000 7f427d7fa640 10 client.?.objecter ms_handle_connect 0x7f42800a98f0 2024-03-25T18:16:58.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.359+0000 7f427d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.364+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827326843 0 0) 0x7f42800ac150 con 0x7f42801430c0 2024-03-25T18:16:58.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.364+0000 7f427d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.364+0000 7f427d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.364+0000 7f427d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.364+0000 7f427d7fa640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:16:58.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.364+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4264004eb0 con 0x7f42801430c0 2024-03-25T18:16:58.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.367+0000 7f427d7fa640 10 client.?.objecter ms_handle_connect 0x7f428013f710 2024-03-25T18:16:58.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.367+0000 7f427d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.368+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747684055 0 0) 0x7f4280139330 con 0x7f42800a98f0 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.368+0000 7f427d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.368+0000 7f427d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.368+0000 7f427d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.368+0000 7f427d7fa640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:16:58.369 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.368+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42800ac150 con 0x7f42800a98f0 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.375+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423577728 0 0) 0x7f42800abe70 con 0x7f428013f710 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.375+0000 7f427d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.375+0000 7f427d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.375+0000 7f427d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.375+0000 7f427d7fa640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:16:58.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.375+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4280139330 con 0x7f428013f710 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2370715861 0 0) 0x7f4264004eb0 con 0x7f42801430c0 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42800abe70 con 0x7f42801430c0 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4278001c70 con 0x7f42801430c0 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 10 monclient(hunting): dump: 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:58.378 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:58.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.377+0000 7f427d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2534191319 0 0) 0x7f42800abe70 con 0x7f42801430c0 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 >> v1:172.21.15.161:6789/0 conn(0x7f428013f710 legacy=0x7f42800ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 >> v1:172.21.15.132:6789/0 conn(0x7f42800a98f0 legacy=0x7f42801389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 1 monclient: found mon.c 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:58.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42800acc80 con 0x7f42801430c0 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f427d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 10 monclient: _renew_subs 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4280137e30 con 0x7f42801430c0 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 10 monclient: _renew_subs 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4280136880 con 0x7f42801430c0 2024-03-25T18:16:58.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.390+0000 7f4287159640 1 librados: init done 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.391+0000 7f4287159640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.391+0000 7f4287159640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.391+0000 7f4287159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:58.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.391+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42800627b0 con 0x7f42801430c0 2024-03-25T18:16:58.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4278003970 con 0x7f42801430c0 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4278003d80 con 0x7f42801430c0 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427d7fa640 10 monclient: dump: 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:16:58.689 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:16:58.690 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:16:58.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.687+0000 7f427effd640 4 set_mon_vals no callback set 2024-03-25T18:16:58.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.688+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f4278003fa0 con 0x7f42801430c0 2024-03-25T18:16:58.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.689+0000 7f4284ef5640 10 monclient: get_auth_request con 0x7f426406ddf0 auth_method 0 2024-03-25T18:16:58.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.689+0000 7f427d7fa640 10 client.4370.objecter ms_handle_connect 0x7f426406ddf0 2024-03-25T18:16:58.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f4278054ac0 con 0x7f42801430c0 2024-03-25T18:16:58.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 10 client.4370.objecter ms_dispatch 0x7f4280000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:58.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:58.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:58.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T18:16:58.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42780505d0 con 0x7f42801430c0 2024-03-25T18:16:58.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:58.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.694+0000 7f427d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:58.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.946+0000 7f4287159640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:58.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.946+0000 7f4287159640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:58.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.946+0000 7f4287159640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:16:58.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.946+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42801390e0 con 0x7f42801430c0 2024-03-25T18:16:58.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.978+0000 7f427d7fa640 1 -- 172.21.15.132:0/4093008850 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f4278052500 con 0x7f42801430c0 2024-03-25T18:16:58.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.978+0000 7f427d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:58.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.978+0000 7f427d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:58.979 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:16:58.979 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 10 librados: watch_flush enter 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 10 librados: watch_flush exit 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 10 client.4370.objecter successfully canceled tick 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 >> v1:172.21.15.161:6800/411950978 conn(0x7f426406ddf0 legacy=0x7f42640702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 10 monclient: shutdown 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:58.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 >> v1:172.21.15.132:6790/0 conn(0x7f42801430c0 legacy=0x7f42801e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 shutdown_connections 2024-03-25T18:16:58.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 >> 172.21.15.132:0/4093008850 conn(0x7f42800b7ea0 msgr2=0x7f4280074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:58.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 shutdown_connections 2024-03-25T18:16:58.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 -- 172.21.15.132:0/4093008850 wait complete. 2024-03-25T18:16:58.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:16:58.981+0000 7f4287159640 1 librados: shutdown 2024-03-25T18:16:58.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:05.800 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.916+0000 7fe04020d640 10 monclient: get_monmap_and_config 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.916+0000 7fe04020d640 10 monclient: build_initial_monmap 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.916+0000 7fe04020d640 10 monclient: monmap: 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:05.917233+0000 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:05.917233+0000 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:05.917 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.916+0000 7fe04020d640 1 Processor -- start 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 1 -- start start 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient: init 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient(hunting): picked mon.noname-c con 0x7fe0380a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient(hunting): picked mon.noname-a con 0x7fe03813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0381430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe038142520 con 0x7fe03813f710 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0380b28a0 con 0x7fe0380a98f0 2024-03-25T18:17:05.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0380b2a30 con 0x7fe0381430c0 2024-03-25T18:17:05.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:05.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.917+0000 7fe04020d640 10 monclient(hunting): authenticate will time out at 7466.180578s 2024-03-25T18:17:05.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.934+0000 7fe03dfa9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe0380a98f0 0x7fe0380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:40162/0 (socket says 172.21.15.132:40162) 2024-03-25T18:17:05.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.934+0000 7fe03dfa9640 1 -- 172.21.15.132:0/2729528552 learned_addr learned my addr 172.21.15.132:0/2729528552 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:05.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.948+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116752392 0 0) 0x7fe0380b28a0 con 0x7fe0380a98f0 2024-03-25T18:17:05.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.948+0000 7fe03cfa7640 10 monclient(hunting): _negotiate 2024-03-25T18:17:05.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.948+0000 7fe03cfa7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:05.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.948+0000 7fe03cfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:05.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.948+0000 7fe03cfa7640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:17:05.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.948+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe024003520 con 0x7fe0380a98f0 2024-03-25T18:17:05.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335546037 0 0) 0x7fe038142520 con 0x7fe03813f710 2024-03-25T18:17:05.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient(hunting): _negotiate 2024-03-25T18:17:05.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0380b28a0 con 0x7fe03813f710 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2246248947 0 0) 0x7fe024003520 con 0x7fe0380a98f0 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 >> v1:172.21.15.132:6790/0 conn(0x7fe0381430c0 legacy=0x7fe03814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 >> v1:172.21.15.132:6789/0 conn(0x7fe03813f710 legacy=0x7fe0381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: _finish_hunting 0 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 monclient: found mon.noname-c 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0380b44c0 con 0x7fe0380a98f0 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: _finish_auth 0 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: _check_auth_tickets 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:05.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe034001610 con 0x7fe0380a98f0 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe04020d640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe04020d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe03cfa7640 10 monclient: dump: 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:05.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.968+0000 7fe04020d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:05.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe034001770 con 0x7fe0380a98f0 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe03cfa7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe03cfa7640 1 -- 172.21.15.132:0/2729528552 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe034001bc0 con 0x7fe0380a98f0 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe04020d640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe03cfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe04020d640 4 set_mon_vals no callback set 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe03cfa7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.977+0000 7fe03cfa7640 10 monclient: dump: 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:05.979 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:05.980 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:05.980 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 10 monclient: shutdown 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 1 -- 172.21.15.132:0/2729528552 >> v1:172.21.15.161:6789/0 conn(0x7fe0380a98f0 legacy=0x7fe0380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 1 -- 172.21.15.132:0/2729528552 shutdown_connections 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 1 -- 172.21.15.132:0/2729528552 >> 172.21.15.132:0/2729528552 conn(0x7fe0380b7ea0 msgr2=0x7fe0380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 1 -- 172.21.15.132:0/2729528552 shutdown_connections 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.978+0000 7fe04020d640 1 -- 172.21.15.132:0/2729528552 wait complete. 2024-03-25T18:17:05.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 10 monclient: build_initial_monmap 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 10 monclient: monmap: 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 1 librados: starting msgr at 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 1 librados: starting objecter 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 1 Processor -- start 2024-03-25T18:17:05.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 1 -- start start 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 1 librados: setting wanted keys 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 1 librados: calling monclient init 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.979+0000 7fe04020d640 10 monclient: init 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient(hunting): picked mon.noname-b con 0x7fe0380a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient(hunting): picked mon.noname-c con 0x7fe03813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient(hunting): picked mon.noname-a con 0x7fe0381430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0381dac30 con 0x7fe0381430c0 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0381319b0 con 0x7fe0380a98f0 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe038137010 con 0x7fe03813f710 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:05.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:05.980+0000 7fe04020d640 10 monclient(hunting): authenticate will time out at 7466.243209s 2024-03-25T18:17:06.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.000+0000 7fe03dfa9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe0380a98f0 0x7fe0381db0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:40226/0 (socket says 172.21.15.132:40226) 2024-03-25T18:17:06.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.000+0000 7fe03dfa9640 1 -- 172.21.15.132:0/4058905285 learned_addr learned my addr 172.21.15.132:0/4058905285 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:06.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.000+0000 7fe02e7fc640 10 client.?.objecter ms_handle_connect 0x7fe0381430c0 2024-03-25T18:17:06.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.000+0000 7fe02e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.001+0000 7fe02e7fc640 10 client.?.objecter ms_handle_connect 0x7fe0380a98f0 2024-03-25T18:17:06.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.001+0000 7fe02e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.006 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104654601 0 0) 0x7fe0381dac30 con 0x7fe0381430c0 2024-03-25T18:17:06.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:17:06.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c004eb0 con 0x7fe0381430c0 2024-03-25T18:17:06.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223242953 0 0) 0x7fe0381319b0 con 0x7fe0380a98f0 2024-03-25T18:17:06.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:17:06.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.006+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0381dac30 con 0x7fe0380a98f0 2024-03-25T18:17:06.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.016+0000 7fe02e7fc640 10 client.?.objecter ms_handle_connect 0x7fe03813f710 2024-03-25T18:17:06.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.016+0000 7fe02e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4233878192 0 0) 0x7fe0381dac30 con 0x7fe0380a98f0 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0381319b0 con 0x7fe0380a98f0 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe034001b40 con 0x7fe0380a98f0 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.022 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:06.023 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:06.023 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:06.023 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:06.023 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:06.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.021+0000 7fe02e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:06.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.022+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3129548811 0 0) 0x7fe01c004eb0 con 0x7fe0381430c0 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.022+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0381dac30 con 0x7fe0381430c0 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.022+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe030001b20 con 0x7fe0381430c0 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.022+0000 7fe02e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.022+0000 7fe02e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.022+0000 7fe02e7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:06.024 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.023+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429513738 0 0) 0x7fe038137010 con 0x7fe03813f710 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.023+0000 7fe02e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.023+0000 7fe02e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.023+0000 7fe02e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.023+0000 7fe02e7fc640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:17:06.025 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.023+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe01c004eb0 con 0x7fe03813f710 2024-03-25T18:17:06.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1201652160 0 0) 0x7fe0381dac30 con 0x7fe0381430c0 2024-03-25T18:17:06.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 >> v1:172.21.15.132:6790/0 conn(0x7fe03813f710 legacy=0x7fe038130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 >> v1:172.21.15.161:6789/0 conn(0x7fe0380a98f0 legacy=0x7fe0381db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.029 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 1 monclient: found mon.a 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0381351c0 con 0x7fe0381430c0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe02e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 10 monclient: _renew_subs 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0381db7d0 con 0x7fe0381430c0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 10 monclient: _renew_subs 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe038139f00 con 0x7fe0381430c0 2024-03-25T18:17:06.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.028+0000 7fe04020d640 1 librados: init done 2024-03-25T18:17:06.031 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.029+0000 7fe04020d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.031 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.029+0000 7fe04020d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.031 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.029+0000 7fe04020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:06.031 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.029+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe038136e50 con 0x7fe0381430c0 2024-03-25T18:17:06.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe030002050 con 0x7fe0381430c0 2024-03-25T18:17:06.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe030003b10 con 0x7fe0381430c0 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02e7fc640 10 monclient: dump: 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:06.337 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:06.338 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:06.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.335+0000 7fe02ffff640 4 set_mon_vals no callback set 2024-03-25T18:17:06.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.336+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fe030019ae0 con 0x7fe0381430c0 2024-03-25T18:17:06.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.337+0000 7fe03dfa9640 10 monclient: get_auth_request con 0x7fe01c06def0 auth_method 0 2024-03-25T18:17:06.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.337+0000 7fe02e7fc640 10 client.4386.objecter ms_handle_connect 0x7fe01c06def0 2024-03-25T18:17:06.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.342+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fe030053830 con 0x7fe0381430c0 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.342+0000 7fe02e7fc640 10 client.4386.objecter ms_dispatch 0x7fe038000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.342+0000 7fe02e7fc640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.342+0000 7fe02e7fc640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.342+0000 7fe02e7fc640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.342+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe03004f170 con 0x7fe0381430c0 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.343+0000 7fe02e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.343+0000 7fe02e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:06.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.600+0000 7fe04020d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.600+0000 7fe04020d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.600+0000 7fe04020d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:06.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.600+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0380627b0 con 0x7fe0381430c0 2024-03-25T18:17:06.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.620+0000 7fe02e7fc640 1 -- 172.21.15.132:0/4058905285 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fe030051270 con 0x7fe0381430c0 2024-03-25T18:17:06.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.620+0000 7fe02e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:06.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.620+0000 7fe02e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:06.621 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:06.621 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 10 librados: watch_flush enter 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 10 librados: watch_flush exit 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 >> v1:172.21.15.161:6800/411950978 conn(0x7fe01c06def0 legacy=0x7fe01c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 10 monclient: shutdown 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 >> v1:172.21.15.132:6789/0 conn(0x7fe0381430c0 legacy=0x7fe0381312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 shutdown_connections 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 >> 172.21.15.132:0/4058905285 conn(0x7fe0380b7ea0 msgr2=0x7fe038074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 shutdown_connections 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 -- 172.21.15.132:0/4058905285 wait complete. 2024-03-25T18:17:06.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:06.622+0000 7fe04020d640 1 librados: shutdown 2024-03-25T18:17:06.636 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:13.443 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.560+0000 7f582f4cf640 10 monclient: get_monmap_and_config 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.560+0000 7f582f4cf640 10 monclient: build_initial_monmap 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.560+0000 7f582f4cf640 10 monclient: monmap: 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:13.561231+0000 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:13.561231+0000 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:13.561 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:13.562 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:13.562 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:13.562 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:13.562 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:13.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.560+0000 7f582f4cf640 1 Processor -- start 2024-03-25T18:17:13.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 1 -- start start 2024-03-25T18:17:13.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient: init 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient(hunting): picked mon.noname-b con 0x7f58280ae6e0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient(hunting): picked mon.noname-a con 0x7f582806f080 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient(hunting): picked mon.noname-c con 0x7f58280704e0 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280718d0 con 0x7f582806f080 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582806fa30 con 0x7f58280704e0 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5828070090 con 0x7f58280ae6e0 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:13.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.561+0000 7f582f4cf640 10 monclient(hunting): authenticate will time out at 7473.824645s 2024-03-25T18:17:13.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.573+0000 7f582da6c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f58280704e0 0x7f582814a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:48264/0 (socket says 172.21.15.132:48264) 2024-03-25T18:17:13.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.573+0000 7f582da6c640 1 -- 172.21.15.132:0/1926084493 learned_addr learned my addr 172.21.15.132:0/1926084493 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:13.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.581+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044230935 0 0) 0x7f582806fa30 con 0x7f58280704e0 2024-03-25T18:17:13.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.581+0000 7f581ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.581+0000 7f581ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.581+0000 7f581ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.581+0000 7f581ffff640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:17:13.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.581+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5814003520 con 0x7f58280704e0 2024-03-25T18:17:13.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.583+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990989135 0 0) 0x7f58280718d0 con 0x7f582806f080 2024-03-25T18:17:13.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.583+0000 7f581ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.583+0000 7f581ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.583+0000 7f581ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.583+0000 7f581ffff640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:17:13.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.583+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f582806fa30 con 0x7f582806f080 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3736225998 0 0) 0x7f582806fa30 con 0x7f582806f080 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 >> v1:172.21.15.132:6790/0 conn(0x7f58280ae6e0 legacy=0x7f582813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 >> v1:172.21.15.161:6789/0 conn(0x7f58280704e0 legacy=0x7f582814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 1 monclient: found mon.noname-a 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:17:13.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582806f510 con 0x7f582806f080 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5818001670 con 0x7f582806f080 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f581ffff640 10 monclient: dump: 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:13.592 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:13.593 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:13.593 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:13.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f582f4cf640 5 monclient: authenticate success, global_id 4389 2024-03-25T18:17:13.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.590+0000 7f582f4cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58180017d0 con 0x7f582806f080 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f581ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f581ffff640 1 -- 172.21.15.132:0/1926084493 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5818001be0 con 0x7f582806f080 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f581ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f581ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f581ffff640 10 monclient: dump: 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:13.599 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f582f4cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:13.600 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.598+0000 7f582f4cf640 4 set_mon_vals no callback set 2024-03-25T18:17:13.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 10 monclient: shutdown 2024-03-25T18:17:13.601 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 1 -- 172.21.15.132:0/1926084493 >> v1:172.21.15.132:6789/0 conn(0x7f582806f080 legacy=0x7f5828071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 1 -- 172.21.15.132:0/1926084493 shutdown_connections 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 1 -- 172.21.15.132:0/1926084493 >> 172.21.15.132:0/1926084493 conn(0x7f582806b630 msgr2=0x7f58280b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 1 -- 172.21.15.132:0/1926084493 shutdown_connections 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.599+0000 7f582f4cf640 1 -- 172.21.15.132:0/1926084493 wait complete. 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.600+0000 7f582f4cf640 10 monclient: build_initial_monmap 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.600+0000 7f582f4cf640 10 monclient: monmap: 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:13.602 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 1 librados: starting msgr at 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 1 librados: starting objecter 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 1 Processor -- start 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 1 -- start start 2024-03-25T18:17:13.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 1 librados: setting wanted keys 2024-03-25T18:17:13.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 1 librados: calling monclient init 2024-03-25T18:17:13.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 10 monclient: init 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 10 monclient(hunting): picked mon.noname-c con 0x7f58280704e0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 10 monclient(hunting): picked mon.noname-b con 0x7f58280ae6e0 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.601+0000 7f582f4cf640 10 monclient(hunting): picked mon.noname-a con 0x7f5828142c90 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.602+0000 7f582f4cf640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280a9470 con 0x7f5828142c90 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.602+0000 7f582f4cf640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280ab490 con 0x7f58280ae6e0 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.602+0000 7f582f4cf640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280ab770 con 0x7f58280704e0 2024-03-25T18:17:13.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.602+0000 7f582f4cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:13.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.602+0000 7f582f4cf640 10 monclient(hunting): authenticate will time out at 7473.864795s 2024-03-25T18:17:13.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.613+0000 7f582ca6a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f58280ae6e0 0x7f58280abb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:48300/0 (socket says 172.21.15.132:48300) 2024-03-25T18:17:13.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.613+0000 7f582ca6a640 1 -- 172.21.15.132:0/2999844777 learned_addr learned my addr 172.21.15.132:0/2999844777 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:13.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.613+0000 7f581dffb640 10 client.?.objecter ms_handle_connect 0x7f58280ae6e0 2024-03-25T18:17:13.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.613+0000 7f581dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.615+0000 7f581dffb640 10 client.?.objecter ms_handle_connect 0x7f5828142c90 2024-03-25T18:17:13.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.615+0000 7f581dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.618+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086992453 0 0) 0x7f58280a9470 con 0x7f5828142c90 2024-03-25T18:17:13.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.618+0000 7f581dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.618+0000 7f581dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.618+0000 7f581dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.618+0000 7f581dffb640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:17:13.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.618+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c004eb0 con 0x7f5828142c90 2024-03-25T18:17:13.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.625+0000 7f581dffb640 10 client.?.objecter ms_handle_connect 0x7f58280704e0 2024-03-25T18:17:13.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.625+0000 7f581dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:13.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 775449840 0 0) 0x7f580c004eb0 con 0x7f5828142c90 2024-03-25T18:17:13.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58280a9470 con 0x7f5828142c90 2024-03-25T18:17:13.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5820001cd0 con 0x7f5828142c90 2024-03-25T18:17:13.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.630 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 10 monclient(hunting): dump: 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:13.631 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.629+0000 7f581dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:13.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.633+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461033641 0 0) 0x7f58280ab490 con 0x7f58280ae6e0 2024-03-25T18:17:13.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.633+0000 7f581dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:13.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.633+0000 7f581dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:13.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.633+0000 7f581dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:13.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.633+0000 7f581dffb640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:17:13.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.633+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f580c004eb0 con 0x7f58280ae6e0 2024-03-25T18:17:13.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2241421946 0 0) 0x7f58280a9470 con 0x7f5828142c90 2024-03-25T18:17:13.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 >> v1:172.21.15.132:6790/0 conn(0x7f58280704e0 legacy=0x7f582813d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 >> v1:172.21.15.161:6789/0 conn(0x7f58280ae6e0 legacy=0x7f58280abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:13.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:13.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 1 monclient: found mon.a 2024-03-25T18:17:13.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58280ad1b0 con 0x7f5828142c90 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f581dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 5 monclient: authenticate success, global_id 4392 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 10 monclient: _renew_subs 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58280ac9b0 con 0x7f5828142c90 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 10 monclient: _renew_subs 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:13.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f582813ef30 con 0x7f5828142c90 2024-03-25T18:17:13.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.634+0000 7f582f4cf640 1 librados: init done 2024-03-25T18:17:13.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.635+0000 7f582f4cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.635+0000 7f582f4cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.635+0000 7f582f4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:13.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.635+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582813ceb0 con 0x7f5828142c90 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58200020e0 con 0x7f5828142c90 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5820004d00 con 0x7f5828142c90 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 10 monclient: dump: 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:13.935 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:13.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581f7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:13.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.934+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f5820004f20 con 0x7f5828142c90 2024-03-25T18:17:13.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.935+0000 7f582d26b640 10 monclient: get_auth_request con 0x7f580c06dd60 auth_method 0 2024-03-25T18:17:13.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.935+0000 7f581dffb640 10 client.4392.objecter ms_handle_connect 0x7f580c06dd60 2024-03-25T18:17:13.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f5820054a50 con 0x7f5828142c90 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 10 client.4392.objecter ms_dispatch 0x7f5828000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 3 client.4392.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 3 client.4392.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 20 client.4392.objecter dump_active .. 0 homeless 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5820050390 con 0x7f5828142c90 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:13.939+0000 7f581dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:14.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.190+0000 7f582f4cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.190+0000 7f582f4cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.190+0000 7f582f4cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:14.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.190+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58280627b0 con 0x7f5828142c90 2024-03-25T18:17:14.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.238+0000 7f581dffb640 1 -- 172.21.15.132:0/2999844777 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f5820052490 con 0x7f5828142c90 2024-03-25T18:17:14.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.238+0000 7f581dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.238+0000 7f581dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:14.240 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:14.240 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 10 librados: watch_flush enter 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 10 librados: watch_flush exit 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 20 client.4392.objecter shutdown clearing up homeless session... 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 10 client.4392.objecter successfully canceled tick 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 >> v1:172.21.15.161:6800/411950978 conn(0x7f580c06dd60 legacy=0x7f580c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 10 monclient: shutdown 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 >> v1:172.21.15.132:6789/0 conn(0x7f5828142c90 legacy=0x7f58280ac2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 shutdown_connections 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 >> 172.21.15.132:0/2999844777 conn(0x7f582806b630 msgr2=0x7f58280b31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:14.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 shutdown_connections 2024-03-25T18:17:14.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.241+0000 7f582f4cf640 1 -- 172.21.15.132:0/2999844777 wait complete. 2024-03-25T18:17:14.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:14.242+0000 7f582f4cf640 1 librados: shutdown 2024-03-25T18:17:14.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:14.571 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:59.836 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.583 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:59.847 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.606 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:03:59.870 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.620 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:59.862 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.629 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:59.893 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.634 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:59.876 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.641 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:59.905 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.670 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:03:59.911 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.683 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:03:59.947 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:21.061 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:21.179 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.178+0000 7f7ecdee6640 10 monclient: get_monmap_and_config 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.178+0000 7f7ecdee6640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.179+0000 7f7ecdee6640 10 monclient: monmap: 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:21.180051+0000 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:21.180051+0000 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.180 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:21.181 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:21.181 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:21.181 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:21.181 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:21.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 1 Processor -- start 2024-03-25T18:17:21.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 1 -- start start 2024-03-25T18:17:21.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient: init 2024-03-25T18:17:21.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec80b0070 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec813c170 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec806fa90 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b0450 con 0x7f7ec813c170 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8071440 con 0x7f7ec80b0070 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec80b0bf0 con 0x7f7ec806fa90 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.180+0000 7f7ecdee6640 10 monclient(hunting): authenticate will time out at 7481.443524s 2024-03-25T18:17:21.188 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.187+0000 7f7ec77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7ec80b0070 0x7f7ec813ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:52700/0 (socket says 172.21.15.132:52700) 2024-03-25T18:17:21.188 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.187+0000 7f7ec77fe640 1 -- 172.21.15.132:0/2842698610 learned_addr learned my addr 172.21.15.132:0/2842698610 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:21.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.200+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960281688 0 0) 0x7f7ec8071440 con 0x7f7ec80b0070 2024-03-25T18:17:21.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.200+0000 7f7ec67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.200+0000 7f7ec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.200+0000 7f7ec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.200+0000 7f7ec67fc640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:17:21.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.200+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb0003520 con 0x7f7ec80b0070 2024-03-25T18:17:21.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.204+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275080930 0 0) 0x7f7ec80b0bf0 con 0x7f7ec806fa90 2024-03-25T18:17:21.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.204+0000 7f7ec67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.204+0000 7f7ec67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.204+0000 7f7ec67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.204+0000 7f7ec67fc640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:17:21.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.204+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8071440 con 0x7f7ec806fa90 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2047669334 0 0) 0x7f7eb0003520 con 0x7f7ec80b0070 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 >> v1:172.21.15.132:6790/0 conn(0x7f7ec806fa90 legacy=0x7f7ec80615c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 >> v1:172.21.15.132:6789/0 conn(0x7f7ec813c170 legacy=0x7f7ec8141d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec80716b0 con 0x7f7ec80b0070 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7eb8001690 con 0x7f7ec80b0070 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ec67fc640 10 monclient: dump: 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.220 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ecdee6640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:17:21.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.218+0000 7f7ecdee6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:21.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb80017f0 con 0x7f7ec80b0070 2024-03-25T18:17:21.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ec67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ec67fc640 1 -- 172.21.15.132:0/2842698610 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7eb8001c00 con 0x7f7ec80b0070 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ec67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ec67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ec67fc640 10 monclient: dump: 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:21.245 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ecdee6640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.242+0000 7f7ecdee6640 4 set_mon_vals no callback set 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 10 monclient: shutdown 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 1 -- 172.21.15.132:0/2842698610 >> v1:172.21.15.161:6789/0 conn(0x7f7ec80b0070 legacy=0x7f7ec813ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 1 -- 172.21.15.132:0/2842698610 shutdown_connections 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 1 -- 172.21.15.132:0/2842698610 >> 172.21.15.132:0/2842698610 conn(0x7f7ec80b0ef0 msgr2=0x7f7ec80b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 1 -- 172.21.15.132:0/2842698610 shutdown_connections 2024-03-25T18:17:21.246 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.243+0000 7f7ecdee6640 1 -- 172.21.15.132:0/2842698610 wait complete. 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.244+0000 7f7ecdee6640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.244+0000 7f7ecdee6640 10 monclient: monmap: 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:21.248 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.244+0000 7f7ecdee6640 1 librados: starting msgr at 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.244+0000 7f7ecdee6640 1 librados: starting objecter 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.244+0000 7f7ecdee6640 1 Processor -- start 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 1 -- start start 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 1 librados: setting wanted keys 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 1 librados: calling monclient init 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient: init 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec806fa90 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:21.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec80b0070 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:21.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec813c170 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:21.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8143f50 con 0x7f7ec80b0070 2024-03-25T18:17:21.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec8136f90 con 0x7f7ec806fa90 2024-03-25T18:17:21.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec81dac40 con 0x7f7ec813c170 2024-03-25T18:17:21.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.245+0000 7f7ecdee6640 10 monclient(hunting): authenticate will time out at 7481.508396s 2024-03-25T18:17:21.253 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.252+0000 7f7ec7fff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f7ec813c170 0x7f7ec8136880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:59092/0 (socket says 172.21.15.132:59092) 2024-03-25T18:17:21.253 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.252+0000 7f7ec7fff640 1 -- 172.21.15.132:0/96978734 learned_addr learned my addr 172.21.15.132:0/96978734 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:21.253 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.253+0000 7f7eaffff640 10 client.?.objecter ms_handle_connect 0x7f7ec813c170 2024-03-25T18:17:21.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.253+0000 7f7eaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.255+0000 7f7eaffff640 10 client.?.objecter ms_handle_connect 0x7f7ec806fa90 2024-03-25T18:17:21.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.255+0000 7f7eaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.259+0000 7f7eaffff640 10 client.?.objecter ms_handle_connect 0x7f7ec80b0070 2024-03-25T18:17:21.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.259+0000 7f7eaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.262 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.261+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469363568 0 0) 0x7f7ec81dac40 con 0x7f7ec813c170 2024-03-25T18:17:21.262 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.261+0000 7f7eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.261+0000 7f7eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.261+0000 7f7eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.261+0000 7f7eaffff640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:17:21.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.261+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eb4004eb0 con 0x7f7ec813c170 2024-03-25T18:17:21.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.273+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551270737 0 0) 0x7f7ec8136f90 con 0x7f7ec806fa90 2024-03-25T18:17:21.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.273+0000 7f7eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.273+0000 7f7eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.273+0000 7f7eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.273+0000 7f7eaffff640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:17:21.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.273+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec81dac40 con 0x7f7ec806fa90 2024-03-25T18:17:21.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.278+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026751325 0 0) 0x7f7ec8143f50 con 0x7f7ec80b0070 2024-03-25T18:17:21.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.278+0000 7f7eaffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.278+0000 7f7eaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.278+0000 7f7eaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.278+0000 7f7eaffff640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:17:21.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.278+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ec8136f90 con 0x7f7ec80b0070 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2179093345 0 0) 0x7f7ec81dac40 con 0x7f7ec806fa90 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ec8143f50 con 0x7f7ec806fa90 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7eb8001b40 con 0x7f7ec806fa90 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 10 monclient(hunting): dump: 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:21.286 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:21.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.285+0000 7f7eaffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2179292108 0 0) 0x7f7ec8143f50 con 0x7f7ec806fa90 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 >> v1:172.21.15.132:6790/0 conn(0x7f7ec813c170 legacy=0x7f7ec8136880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 >> v1:172.21.15.132:6789/0 conn(0x7f7ec80b0070 legacy=0x7f7ec8136170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 1 monclient: found mon.b 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8144230 con 0x7f7ec806fa90 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7eaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 10 monclient: _renew_subs 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec813f5b0 con 0x7f7ec806fa90 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 10 monclient: _renew_subs 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec81db830 con 0x7f7ec806fa90 2024-03-25T18:17:21.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.294+0000 7f7ecdee6640 1 librados: init done 2024-03-25T18:17:21.297 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.295+0000 7f7ecdee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.297 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.295+0000 7f7ecdee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.297 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.295+0000 7f7ecdee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:21.297 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.295+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec8070b60 con 0x7f7ec806fa90 2024-03-25T18:17:21.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7eb8004a90 con 0x7f7ec806fa90 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7eaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7eb8004da0 con 0x7f7ec806fa90 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7eaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7eaffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7eaffff640 10 monclient: dump: 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:21.576 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:21.577 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:21.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.574+0000 7f7ec5ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:21.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.575+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f7eb801ad70 con 0x7f7ec806fa90 2024-03-25T18:17:21.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.576+0000 7f7ec6ffd640 10 monclient: get_auth_request con 0x7f7eb406dc30 auth_method 0 2024-03-25T18:17:21.577 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.576+0000 7f7eaffff640 10 client.4396.objecter ms_handle_connect 0x7f7eb406dc30 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f7eb8054880 con 0x7f7ec806fa90 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 10 client.4396.objecter ms_dispatch 0x7f7ec8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7eb8050390 con 0x7f7ec806fa90 2024-03-25T18:17:21.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.582+0000 7f7eaffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:21.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.785+0000 7f7ecdee6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.785+0000 7f7ecdee6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.785+0000 7f7ecdee6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:21.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.785+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ec80627b0 con 0x7f7ec806fa90 2024-03-25T18:17:21.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.853+0000 7f7eaffff640 1 -- 172.21.15.132:0/96978734 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f7eb80522c0 con 0x7f7ec806fa90 2024-03-25T18:17:21.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.853+0000 7f7eaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:21.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.853+0000 7f7eaffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:21.854 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:21.854 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:21.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 10 librados: watch_flush enter 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 10 librados: watch_flush exit 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 >> v1:172.21.15.161:6800/411950978 conn(0x7f7eb406dc30 legacy=0x7f7eb40700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 10 monclient: shutdown 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 >> v1:172.21.15.161:6789/0 conn(0x7f7ec806fa90 legacy=0x7f7ec81db120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 shutdown_connections 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 >> 172.21.15.132:0/96978734 conn(0x7f7ec80b0ef0 msgr2=0x7f7ec80615c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 shutdown_connections 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 -- 172.21.15.132:0/96978734 wait complete. 2024-03-25T18:17:21.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:21.855+0000 7f7ecdee6640 1 librados: shutdown 2024-03-25T18:17:21.867 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:28.674 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:28.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.788+0000 7fae15033640 10 monclient: get_monmap_and_config 2024-03-25T18:17:28.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.788+0000 7fae15033640 10 monclient: build_initial_monmap 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.788+0000 7fae15033640 10 monclient: monmap: 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:28.789985+0000 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:28.789985+0000 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:28.790 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.789+0000 7fae15033640 1 Processor -- start 2024-03-25T18:17:28.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.789+0000 7fae15033640 1 -- start start 2024-03-25T18:17:28.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.789+0000 7fae15033640 10 monclient: init 2024-03-25T18:17:28.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient(hunting): picked mon.noname-b con 0x7fae100a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient(hunting): picked mon.noname-c con 0x7fae1013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient(hunting): picked mon.noname-a con 0x7fae101430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae10142520 con 0x7fae101430c0 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100b28a0 con 0x7fae1013f710 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae100b2a30 con 0x7fae100a98f0 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:28.792 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.790+0000 7fae15033640 10 monclient(hunting): authenticate will time out at 7489.053196s 2024-03-25T18:17:28.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.797+0000 7fae0e59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fae1013f710 0x7fae101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:43598/0 (socket says 172.21.15.132:43598) 2024-03-25T18:17:28.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.797+0000 7fae0e59c640 1 -- 172.21.15.132:0/2699768191 learned_addr learned my addr 172.21.15.132:0/2699768191 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:28.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.804+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371495230 0 0) 0x7fae10142520 con 0x7fae101430c0 2024-03-25T18:17:28.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.804+0000 7fae0dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.804+0000 7fae0dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.804+0000 7fae0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.804+0000 7fae0dd9b640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:17:28.805 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.804+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf8003520 con 0x7fae101430c0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1987090607 0 0) 0x7fadf8003520 con 0x7fae101430c0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 >> v1:172.21.15.132:6790/0 conn(0x7fae100a98f0 legacy=0x7fae100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 >> v1:172.21.15.161:6789/0 conn(0x7fae1013f710 legacy=0x7fae101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae100b44c0 con 0x7fae101430c0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae08001690 con 0x7fae101430c0 2024-03-25T18:17:28.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae0dd9b640 10 monclient: dump: 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae15033640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:17:28.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.808+0000 7fae15033640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:28.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae080017f0 con 0x7fae101430c0 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae0dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 4 set_mon_vals no callback set 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae0dd9b640 1 -- 172.21.15.132:0/2699768191 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae08001c00 con 0x7fae101430c0 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae0dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae0dd9b640 10 monclient: dump: 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:28.820 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:28.821 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:28.821 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:28.821 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 10 monclient: shutdown 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 1 -- 172.21.15.132:0/2699768191 >> v1:172.21.15.132:6789/0 conn(0x7fae101430c0 legacy=0x7fae1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 1 -- 172.21.15.132:0/2699768191 shutdown_connections 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.819+0000 7fae15033640 1 -- 172.21.15.132:0/2699768191 >> 172.21.15.132:0/2699768191 conn(0x7fae100b7ea0 msgr2=0x7fae100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.820+0000 7fae15033640 1 -- 172.21.15.132:0/2699768191 shutdown_connections 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.820+0000 7fae15033640 1 -- 172.21.15.132:0/2699768191 wait complete. 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 10 monclient: build_initial_monmap 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 10 monclient: monmap: 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:28.822 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 1 librados: starting msgr at 2024-03-25T18:17:28.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 1 librados: starting objecter 2024-03-25T18:17:28.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 1 Processor -- start 2024-03-25T18:17:28.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 1 -- start start 2024-03-25T18:17:28.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 1 librados: setting wanted keys 2024-03-25T18:17:28.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 1 librados: calling monclient init 2024-03-25T18:17:28.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.821+0000 7fae15033640 10 monclient: init 2024-03-25T18:17:28.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient(hunting): picked mon.noname-c con 0x7fae100a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient(hunting): picked mon.noname-b con 0x7fae1013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient(hunting): picked mon.noname-a con 0x7fae101430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae10138b60 con 0x7fae101430c0 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae10137e40 con 0x7fae1013f710 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae101d82f0 con 0x7fae100a98f0 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:28.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.822+0000 7fae15033640 10 monclient(hunting): authenticate will time out at 7489.085611s 2024-03-25T18:17:28.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.829+0000 7fae0ed9d640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fae100a98f0 0x7fae10137730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:48548/0 (socket says 172.21.15.132:48548) 2024-03-25T18:17:28.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.829+0000 7fae0ed9d640 1 -- 172.21.15.132:0/1027307159 learned_addr learned my addr 172.21.15.132:0/1027307159 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:28.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.838+0000 7fadf77fe640 10 client.?.objecter ms_handle_connect 0x7fae100a98f0 2024-03-25T18:17:28.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.838+0000 7fadf77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:28.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.838+0000 7fadf77fe640 10 client.?.objecter ms_handle_connect 0x7fae101430c0 2024-03-25T18:17:28.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.838+0000 7fadf77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:28.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.839+0000 7fadf77fe640 10 client.?.objecter ms_handle_connect 0x7fae1013f710 2024-03-25T18:17:28.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.839+0000 7fadf77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564710392 0 0) 0x7fae10137e40 con 0x7fae1013f710 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadfc004eb0 con 0x7fae1013f710 2024-03-25T18:17:28.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678987597 0 0) 0x7fae10138b60 con 0x7fae101430c0 2024-03-25T18:17:28.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:17:28.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.860+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae10137e40 con 0x7fae101430c0 2024-03-25T18:17:28.871 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.870+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267002566 0 0) 0x7fae101d82f0 con 0x7fae100a98f0 2024-03-25T18:17:28.871 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.870+0000 7fadf77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:28.871 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.870+0000 7fadf77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:28.871 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.870+0000 7fadf77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:28.871 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.870+0000 7fadf77fe640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:17:28.871 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.870+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae10138b60 con 0x7fae100a98f0 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 824474374 0 0) 0x7fae10137e40 con 0x7fae101430c0 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae101d82f0 con 0x7fae101430c0 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae08001b00 con 0x7fae101430c0 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 10 monclient(hunting): dump: 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:28.885 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.884+0000 7fadf77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:28.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.893+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 853943455 0 0) 0x7fadfc004eb0 con 0x7fae1013f710 2024-03-25T18:17:28.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.893+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae10137e40 con 0x7fae1013f710 2024-03-25T18:17:28.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.893+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae04001930 con 0x7fae1013f710 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.893+0000 7fadf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.893+0000 7fadf77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.893+0000 7fadf77fe640 10 monclient(hunting): dump: 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:28.895 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:28.896 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:28.896 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.901+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299246291 0 0) 0x7fae10138b60 con 0x7fae100a98f0 2024-03-25T18:17:28.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.901+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadfc004eb0 con 0x7fae100a98f0 2024-03-25T18:17:28.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.901+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae00001c70 con 0x7fae100a98f0 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.901+0000 7fadf77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.901+0000 7fadf77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.901+0000 7fadf77fe640 10 monclient(hunting): dump: 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:28.903 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:28.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3135682048 0 0) 0x7fae101d82f0 con 0x7fae101430c0 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 >> v1:172.21.15.132:6790/0 conn(0x7fae100a98f0 legacy=0x7fae10137730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 >> v1:172.21.15.161:6789/0 conn(0x7fae1013f710 legacy=0x7fae101d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 1 monclient: found mon.a 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae10134e70 con 0x7fae101430c0 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fadf77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 10 monclient: _renew_subs 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:28.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae101350c0 con 0x7fae101430c0 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 10 monclient: _renew_subs 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae10134800 con 0x7fae101430c0 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.904+0000 7fae15033640 1 librados: init done 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.905+0000 7fae15033640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:28.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.905+0000 7fae15033640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:28.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.905+0000 7fae15033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:28.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:28.905+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1014f330 con 0x7fae101430c0 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae08001cc0 con 0x7fae101430c0 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fadf77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae080038a0 con 0x7fae101430c0 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fadf77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fadf77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fadf77fe640 10 monclient: dump: 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:29.219 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.218+0000 7fae0d59a640 4 set_mon_vals no callback set 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.219+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fae08019870 con 0x7fae101430c0 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.219+0000 7fae0ed9d640 10 monclient: get_auth_request con 0x7fadfc06e0f0 auth_method 0 2024-03-25T18:17:29.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.220+0000 7fadf77fe640 10 client.4404.objecter ms_handle_connect 0x7fadfc06e0f0 2024-03-25T18:17:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fae080533a0 con 0x7fae101430c0 2024-03-25T18:17:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 10 client.4404.objecter ms_dispatch 0x7fae10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:17:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae0804eeb0 con 0x7fae101430c0 2024-03-25T18:17:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.226+0000 7fadf77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:29.451 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.450+0000 7fae15033640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.451 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.450+0000 7fae15033640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.451 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.450+0000 7fae15033640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:29.451 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.450+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae10074cc0 con 0x7fae101430c0 2024-03-25T18:17:29.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.494+0000 7fadf77fe640 1 -- 172.21.15.132:0/1027307159 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fae08050de0 con 0x7fae101430c0 2024-03-25T18:17:29.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.494+0000 7fadf77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:29.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.494+0000 7fadf77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:29.495 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:29.495 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.496+0000 7fae15033640 10 librados: watch_flush enter 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.496+0000 7fae15033640 10 librados: watch_flush exit 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.496+0000 7fae15033640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.496+0000 7fae15033640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 >> v1:172.21.15.161:6800/411950978 conn(0x7fadfc06e0f0 legacy=0x7fadfc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 10 monclient: shutdown 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 >> v1:172.21.15.132:6789/0 conn(0x7fae101430c0 legacy=0x7fae10133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 shutdown_connections 2024-03-25T18:17:29.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 >> 172.21.15.132:0/1027307159 conn(0x7fae100b7ea0 msgr2=0x7fae10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:29.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 shutdown_connections 2024-03-25T18:17:29.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 -- 172.21.15.132:0/1027307159 wait complete. 2024-03-25T18:17:29.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:29.497+0000 7fae15033640 1 librados: shutdown 2024-03-25T18:17:29.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:36.317 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:36.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.429+0000 7f2dcf0fb640 10 monclient: get_monmap_and_config 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.429+0000 7f2dcf0fb640 10 monclient: build_initial_monmap 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.429+0000 7f2dcf0fb640 10 monclient: monmap: 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:36.430981+0000 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:36.430981+0000 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:36.431 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.430+0000 7f2dcf0fb640 1 Processor -- start 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 1 -- start start 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient: init 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc80a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc81430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8142520 con 0x7f2dc813f710 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80b28a0 con 0x7f2dc81430c0 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80b2a30 con 0x7f2dc80a98f0 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:36.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.431+0000 7f2dcf0fb640 10 monclient(hunting): authenticate will time out at 7496.694620s 2024-03-25T18:17:36.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.441+0000 7f2dcd698640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2dc81430c0 0x7f2dc814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:34068/0 (socket says 172.21.15.132:34068) 2024-03-25T18:17:36.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.441+0000 7f2dcd698640 1 -- 172.21.15.132:0/992839254 learned_addr learned my addr 172.21.15.132:0/992839254 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:36.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.451+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010118325 0 0) 0x7f2dc80b28a0 con 0x7f2dc81430c0 2024-03-25T18:17:36.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.451+0000 7f2dc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.451+0000 7f2dc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.451+0000 7f2dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.451+0000 7f2dc77fe640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:17:36.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.451+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db8003520 con 0x7f2dc81430c0 2024-03-25T18:17:36.459 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.458+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429384839 0 0) 0x7f2dc80b2a30 con 0x7f2dc80a98f0 2024-03-25T18:17:36.459 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.458+0000 7f2dc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.458+0000 7f2dc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.458+0000 7f2dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.458+0000 7f2dc77fe640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:17:36.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.458+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc80b28a0 con 0x7f2dc80a98f0 2024-03-25T18:17:36.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.466+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4038198798 0 0) 0x7f2dc8142520 con 0x7f2dc813f710 2024-03-25T18:17:36.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.466+0000 7f2dc77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.466+0000 7f2dc77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.466+0000 7f2dc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.466+0000 7f2dc77fe640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:17:36.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.466+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc80b2a30 con 0x7f2dc813f710 2024-03-25T18:17:36.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129128383 0 0) 0x7f2db8003520 con 0x7f2dc81430c0 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 >> v1:172.21.15.132:6790/0 conn(0x7f2dc80a98f0 legacy=0x7f2dc80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 >> v1:172.21.15.132:6789/0 conn(0x7f2dc813f710 legacy=0x7f2dc81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 1 monclient: found mon.noname-c 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc80b44c0 con 0x7f2dc81430c0 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc0001a10 con 0x7f2dc81430c0 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:36.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dc77fe640 10 monclient: dump: 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dcf0fb640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:17:36.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.475+0000 7f2dcf0fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc0001b70 con 0x7f2dc81430c0 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dc77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dc77fe640 1 -- 172.21.15.132:0/992839254 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc0001f80 con 0x7f2dc81430c0 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dc77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dc77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dc77fe640 10 monclient: dump: 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.485 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.486 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:36.486 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:36.486 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:36.486 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dcf0fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:36.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.483+0000 7f2dcf0fb640 4 set_mon_vals no callback set 2024-03-25T18:17:36.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 10 monclient: shutdown 2024-03-25T18:17:36.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:36.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/992839254 >> v1:172.21.15.161:6789/0 conn(0x7f2dc81430c0 legacy=0x7f2dc814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/992839254 shutdown_connections 2024-03-25T18:17:36.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/992839254 >> 172.21.15.132:0/992839254 conn(0x7f2dc80b7ea0 msgr2=0x7f2dc80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/992839254 shutdown_connections 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.484+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/992839254 wait complete. 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.485+0000 7f2dcf0fb640 10 monclient: build_initial_monmap 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.485+0000 7f2dcf0fb640 10 monclient: monmap: 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:36.489 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.485+0000 7f2dcf0fb640 1 librados: starting msgr at 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.485+0000 7f2dcf0fb640 1 librados: starting objecter 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.485+0000 7f2dcf0fb640 1 Processor -- start 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 1 -- start start 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 1 librados: setting wanted keys 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 1 librados: calling monclient init 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient: init 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc80a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:36.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc81430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:36.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8137190 con 0x7f2dc81430c0 2024-03-25T18:17:36.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc81d41e0 con 0x7f2dc813f710 2024-03-25T18:17:36.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc81d4430 con 0x7f2dc80a98f0 2024-03-25T18:17:36.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.486+0000 7f2dcf0fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:36.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.487+0000 7f2dcf0fb640 10 monclient(hunting): authenticate will time out at 7496.749718s 2024-03-25T18:17:36.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.502+0000 7f2dc7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2dc813f710 0x7f2dc81352e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:34108/0 (socket says 172.21.15.132:34108) 2024-03-25T18:17:36.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.502+0000 7f2dc7fff640 1 -- 172.21.15.132:0/321297766 learned_addr learned my addr 172.21.15.132:0/321297766 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:36.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.506+0000 7f2dc57fa640 10 client.?.objecter ms_handle_connect 0x7f2dc80a98f0 2024-03-25T18:17:36.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.506+0000 7f2dc57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.508+0000 7f2dc57fa640 10 client.?.objecter ms_handle_connect 0x7f2dc813f710 2024-03-25T18:17:36.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.508+0000 7f2dc57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.513+0000 7f2dc57fa640 10 client.?.objecter ms_handle_connect 0x7f2dc81430c0 2024-03-25T18:17:36.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.513+0000 7f2dc57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:36.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.518+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532401400 0 0) 0x7f2dc8137190 con 0x7f2dc81430c0 2024-03-25T18:17:36.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.518+0000 7f2dc57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.518+0000 7f2dc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.518+0000 7f2dc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.518+0000 7f2dc57fa640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:17:36.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.518+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac004eb0 con 0x7f2dc81430c0 2024-03-25T18:17:36.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.522+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939642342 0 0) 0x7f2dc81d4430 con 0x7f2dc80a98f0 2024-03-25T18:17:36.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.522+0000 7f2dc57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.522+0000 7f2dc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.522+0000 7f2dc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.522+0000 7f2dc57fa640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:17:36.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.522+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc8137190 con 0x7f2dc80a98f0 2024-03-25T18:17:36.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159891815 0 0) 0x7f2dac004eb0 con 0x7f2dc81430c0 2024-03-25T18:17:36.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc81d4430 con 0x7f2dc81430c0 2024-03-25T18:17:36.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc0001e80 con 0x7f2dc81430c0 2024-03-25T18:17:36.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): dump: 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 664770780 0 0) 0x7f2dc81d41e0 con 0x7f2dc813f710 2024-03-25T18:17:36.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:36.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:36.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:36.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:17:36.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.531+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dac004eb0 con 0x7f2dc813f710 2024-03-25T18:17:36.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.536+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3663918146 0 0) 0x7f2dc8137190 con 0x7f2dc80a98f0 2024-03-25T18:17:36.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.536+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc81d41e0 con 0x7f2dc80a98f0 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.536+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dbc001970 con 0x7f2dc80a98f0 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.536+0000 7f2dc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.536+0000 7f2dc57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.536+0000 7f2dc57fa640 10 monclient(hunting): dump: 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:36.538 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.538+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1517355196 0 0) 0x7f2dac004eb0 con 0x7f2dc813f710 2024-03-25T18:17:36.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.538+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc8137190 con 0x7f2dc813f710 2024-03-25T18:17:36.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.540+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2db4001970 con 0x7f2dc813f710 2024-03-25T18:17:36.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.540+0000 7f2dc57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.540+0000 7f2dc57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:36.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.540+0000 7f2dc57fa640 10 monclient(hunting): dump: 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:36.542 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 543169176 0 0) 0x7f2dc81d4430 con 0x7f2dc81430c0 2024-03-25T18:17:36.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 >> v1:172.21.15.132:6790/0 conn(0x7f2dc80a98f0 legacy=0x7f2dc81d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 >> v1:172.21.15.161:6789/0 conn(0x7f2dc813f710 legacy=0x7f2dc81352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:36.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:36.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 1 monclient: found mon.a 2024-03-25T18:17:36.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc813bf90 con 0x7f2dc81430c0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dc57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 10 monclient: _renew_subs 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc81d5060 con 0x7f2dc81430c0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 10 monclient: _renew_subs 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc8139f00 con 0x7f2dc81430c0 2024-03-25T18:17:36.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.545+0000 7f2dcf0fb640 1 librados: init done 2024-03-25T18:17:36.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.546+0000 7f2dcf0fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.546+0000 7f2dcf0fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.546+0000 7f2dcf0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:36.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.546+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80627b0 con 0x7f2dc81430c0 2024-03-25T18:17:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc0002020 con 0x7f2dc81430c0 2024-03-25T18:17:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2dc0004fb0 con 0x7f2dc81430c0 2024-03-25T18:17:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc57fa640 10 monclient: dump: 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:36.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.842+0000 7f2dc6ffd640 4 set_mon_vals no callback set 2024-03-25T18:17:36.845 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.843+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f2dc001af80 con 0x7f2dc81430c0 2024-03-25T18:17:36.845 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.844+0000 7f2dcce97640 10 monclient: get_auth_request con 0x7f2dac06e0a0 auth_method 0 2024-03-25T18:17:36.845 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.844+0000 7f2dc57fa640 10 client.4410.objecter ms_handle_connect 0x7f2dac06e0a0 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f2dc0054b00 con 0x7f2dc81430c0 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 10 client.4410.objecter ms_dispatch 0x7f2dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dc0050440 con 0x7f2dc81430c0 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:36.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:36.849+0000 7f2dc57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:37.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.079+0000 7f2dcf0fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.079+0000 7f2dcf0fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.079+0000 7f2dcf0fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:17:37.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.079+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dc81d45c0 con 0x7f2dc81430c0 2024-03-25T18:17:37.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.153+0000 7f2dc57fa640 1 -- 172.21.15.132:0/321297766 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f2dc0052540 con 0x7f2dc81430c0 2024-03-25T18:17:37.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.153+0000 7f2dc57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.153+0000 7f2dc57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:37.154 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:37.154 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:37.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.155+0000 7f2dcf0fb640 10 librados: watch_flush enter 2024-03-25T18:17:37.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.155+0000 7f2dcf0fb640 10 librados: watch_flush exit 2024-03-25T18:17:37.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T18:17:37.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 10 client.4410.objecter successfully canceled tick 2024-03-25T18:17:37.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 >> v1:172.21.15.161:6800/411950978 conn(0x7f2dac06e0a0 legacy=0x7f2dac070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 10 monclient: shutdown 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 >> v1:172.21.15.132:6789/0 conn(0x7f2dc81430c0 legacy=0x7f2dc8130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 shutdown_connections 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 >> 172.21.15.132:0/321297766 conn(0x7f2dc80b7ea0 msgr2=0x7f2dc8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 shutdown_connections 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 -- 172.21.15.132:0/321297766 wait complete. 2024-03-25T18:17:37.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:37.156+0000 7f2dcf0fb640 1 librados: shutdown 2024-03-25T18:17:37.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:43.975 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:44.090 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.089+0000 7faa227a3640 10 monclient: get_monmap_and_config 2024-03-25T18:17:44.090 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.089+0000 7faa227a3640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.090 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.089+0000 7faa227a3640 10 monclient: monmap: 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:44.090742+0000 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:44.090742+0000 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:44.091 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.090+0000 7faa227a3640 1 Processor -- start 2024-03-25T18:17:44.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.090+0000 7faa227a3640 1 -- start start 2024-03-25T18:17:44.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.090+0000 7faa227a3640 10 monclient: init 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient(hunting): picked mon.noname-a con 0x7faa1c0a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient(hunting): picked mon.noname-c con 0x7faa1c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient(hunting): picked mon.noname-b con 0x7faa1c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c142520 con 0x7faa1c0a98f0 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c0b28a0 con 0x7faa1c13f710 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c0b2a30 con 0x7faa1c1430c0 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.091+0000 7faa227a3640 10 monclient(hunting): authenticate will time out at 7504.354623s 2024-03-25T18:17:44.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.106+0000 7faa20d40640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7faa1c1430c0 0x7faa1c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:40324/0 (socket says 172.21.15.132:40324) 2024-03-25T18:17:44.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.106+0000 7faa20d40640 1 -- 172.21.15.132:0/2568987826 learned_addr learned my addr 172.21.15.132:0/2568987826 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:44.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.119+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181040289 0 0) 0x7faa1c0b28a0 con 0x7faa1c13f710 2024-03-25T18:17:44.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.119+0000 7faa1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.119+0000 7faa1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.119+0000 7faa1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.119+0000 7faa1affd640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:17:44.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.119+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa04003520 con 0x7faa1c13f710 2024-03-25T18:17:44.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.122+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229862648 0 0) 0x7faa1c142520 con 0x7faa1c0a98f0 2024-03-25T18:17:44.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.122+0000 7faa1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.122+0000 7faa1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.122+0000 7faa1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.122+0000 7faa1affd640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:17:44.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.122+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa1c0b28a0 con 0x7faa1c0a98f0 2024-03-25T18:17:44.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.127+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200478522 0 0) 0x7faa1c0b2a30 con 0x7faa1c1430c0 2024-03-25T18:17:44.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.127+0000 7faa1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.127+0000 7faa1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.127+0000 7faa1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.127+0000 7faa1affd640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:17:44.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.127+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa1c142520 con 0x7faa1c1430c0 2024-03-25T18:17:44.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3076288400 0 0) 0x7faa04003520 con 0x7faa1c13f710 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 >> v1:172.21.15.132:6790/0 conn(0x7faa1c1430c0 legacy=0x7faa1c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 >> v1:172.21.15.132:6789/0 conn(0x7faa1c0a98f0 legacy=0x7faa1c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 1 monclient: found mon.noname-c 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0b44c0 con 0x7faa1c13f710 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7faa10001690 con 0x7faa1c13f710 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa227a3640 5 monclient: authenticate success, global_id 4411 2024-03-25T18:17:44.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa227a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa1affd640 10 monclient: dump: 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.130+0000 7faa227a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa100017f0 con 0x7faa1c13f710 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa1affd640 1 -- 172.21.15.132:0/2568987826 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7faa10001c00 con 0x7faa1c13f710 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa1affd640 10 monclient: dump: 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.141 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa227a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:44.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.140+0000 7faa227a3640 4 set_mon_vals no callback set 2024-03-25T18:17:44.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 10 monclient: shutdown 2024-03-25T18:17:44.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 1 -- 172.21.15.132:0/2568987826 >> v1:172.21.15.161:6789/0 conn(0x7faa1c13f710 legacy=0x7faa1c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 1 -- 172.21.15.132:0/2568987826 shutdown_connections 2024-03-25T18:17:44.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 1 -- 172.21.15.132:0/2568987826 >> 172.21.15.132:0/2568987826 conn(0x7faa1c0b7ea0 msgr2=0x7faa1c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 1 -- 172.21.15.132:0/2568987826 shutdown_connections 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.141+0000 7faa227a3640 1 -- 172.21.15.132:0/2568987826 wait complete. 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 10 monclient: monmap: 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 1 librados: starting msgr at 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 1 librados: starting objecter 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 1 Processor -- start 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 1 -- start start 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 1 librados: setting wanted keys 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 1 librados: calling monclient init 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.142+0000 7faa227a3640 10 monclient: init 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient(hunting): picked mon.noname-c con 0x7faa1c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient(hunting): picked mon.noname-b con 0x7faa1c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient(hunting): picked mon.noname-a con 0x7faa1c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c138c10 con 0x7faa1c1430c0 2024-03-25T18:17:44.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c138e60 con 0x7faa1c13f710 2024-03-25T18:17:44.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c139140 con 0x7faa1c0a98f0 2024-03-25T18:17:44.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.143+0000 7faa227a3640 10 monclient(hunting): authenticate will time out at 7504.406298s 2024-03-25T18:17:44.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.148+0000 7faa1b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7faa1c13f710 0x7faa1c132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:47568/0 (socket says 172.21.15.132:47568) 2024-03-25T18:17:44.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.148+0000 7faa1b7fe640 1 -- 172.21.15.132:0/1483075206 learned_addr learned my addr 172.21.15.132:0/1483075206 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:44.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.151+0000 7faa18ff9640 10 client.?.objecter ms_handle_connect 0x7faa1c13f710 2024-03-25T18:17:44.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.152+0000 7faa18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.156+0000 7faa18ff9640 10 client.?.objecter ms_handle_connect 0x7faa1c0a98f0 2024-03-25T18:17:44.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.156+0000 7faa18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.158+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205922433 0 0) 0x7faa1c138e60 con 0x7faa1c13f710 2024-03-25T18:17:44.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.158+0000 7faa18ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.158+0000 7faa18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.158+0000 7faa18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.158+0000 7faa18ff9640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:17:44.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.158+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa08004eb0 con 0x7faa1c13f710 2024-03-25T18:17:44.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.166+0000 7faa18ff9640 10 client.?.objecter ms_handle_connect 0x7faa1c1430c0 2024-03-25T18:17:44.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.166+0000 7faa18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.179+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842044016 0 0) 0x7faa1c139140 con 0x7faa1c0a98f0 2024-03-25T18:17:44.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.179+0000 7faa18ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.179+0000 7faa18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.179+0000 7faa18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.179+0000 7faa18ff9640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:17:44.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.179+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa1c138e60 con 0x7faa1c0a98f0 2024-03-25T18:17:44.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.181+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497190315 0 0) 0x7faa1c138c10 con 0x7faa1c1430c0 2024-03-25T18:17:44.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.181+0000 7faa18ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.181+0000 7faa18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.181+0000 7faa18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.181+0000 7faa18ff9640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:17:44.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.181+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa1c139140 con 0x7faa1c1430c0 2024-03-25T18:17:44.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3894890206 0 0) 0x7faa08004eb0 con 0x7faa1c13f710 2024-03-25T18:17:44.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa1c138c10 con 0x7faa1c13f710 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7faa10001bc0 con 0x7faa1c13f710 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.182+0000 7faa18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:44.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.209+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3947561857 0 0) 0x7faa1c139140 con 0x7faa1c1430c0 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.209+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa08004eb0 con 0x7faa1c1430c0 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.210+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7faa14001bd0 con 0x7faa1c1430c0 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.210+0000 7faa18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.210+0000 7faa18ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.210+0000 7faa18ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:44.211 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1892028732 0 0) 0x7faa1c138c10 con 0x7faa1c13f710 2024-03-25T18:17:44.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 >> v1:172.21.15.132:6790/0 conn(0x7faa1c0a98f0 legacy=0x7faa1c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 >> v1:172.21.15.132:6789/0 conn(0x7faa1c1430c0 legacy=0x7faa1c1e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 1 monclient: found mon.b 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c139920 con 0x7faa1c13f710 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 10 monclient: _renew_subs 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:44.214 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c1382e0 con 0x7faa1c13f710 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 10 monclient: _renew_subs 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c0ab8b0 con 0x7faa1c13f710 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.212+0000 7faa227a3640 1 librados: init done 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.213+0000 7faa227a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.213+0000 7faa227a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.213+0000 7faa227a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:44.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.213+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c137a20 con 0x7faa1c13f710 2024-03-25T18:17:44.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa100011f0 con 0x7faa1c13f710 2024-03-25T18:17:44.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa1a7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7faa10003b00 con 0x7faa1c13f710 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa18ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.506+0000 7faa18ff9640 10 monclient: dump: 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:44.507 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:44.508 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:44.508 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:44.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.507+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7faa10003d20 con 0x7faa1c13f710 2024-03-25T18:17:44.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.508+0000 7faa1bfff640 10 monclient: get_auth_request con 0x7faa0806df70 auth_method 0 2024-03-25T18:17:44.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.508+0000 7faa18ff9640 10 client.4414.objecter ms_handle_connect 0x7faa0806df70 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7faa100548d0 con 0x7faa1c13f710 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 10 client.4414.objecter ms_dispatch 0x7faa1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa100503e0 con 0x7faa1c13f710 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.513 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.513+0000 7faa18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:44.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.744+0000 7faa227a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.744+0000 7faa227a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.744+0000 7faa227a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:44.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.744+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa1c13d010 con 0x7faa1c13f710 2024-03-25T18:17:44.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.795+0000 7faa18ff9640 1 -- 172.21.15.132:0/1483075206 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7faa10052310 con 0x7faa1c13f710 2024-03-25T18:17:44.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.795+0000 7faa18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:44.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.795+0000 7faa18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:44.796 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:44.796 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.797+0000 7faa227a3640 10 librados: watch_flush enter 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 10 librados: watch_flush exit 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 >> v1:172.21.15.161:6800/411950978 conn(0x7faa0806df70 legacy=0x7faa08070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 10 monclient: shutdown 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 >> v1:172.21.15.161:6789/0 conn(0x7faa1c13f710 legacy=0x7faa1c132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 shutdown_connections 2024-03-25T18:17:44.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 >> 172.21.15.132:0/1483075206 conn(0x7faa1c0b7ea0 msgr2=0x7faa1c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 shutdown_connections 2024-03-25T18:17:44.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 -- 172.21.15.132:0/1483075206 wait complete. 2024-03-25T18:17:44.799 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:44.798+0000 7faa227a3640 1 librados: shutdown 2024-03-25T18:17:44.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:51.617 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:51.734 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.733+0000 7f82818d6640 10 monclient: get_monmap_and_config 2024-03-25T18:17:51.734 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.733+0000 7f82818d6640 10 monclient: build_initial_monmap 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.733+0000 7f82818d6640 10 monclient: monmap: 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:51.734796+0000 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:51.734796+0000 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:51.735 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:51.736 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:51.736 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:51.736 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.734+0000 7f82818d6640 1 Processor -- start 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.734+0000 7f82818d6640 1 -- start start 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.734+0000 7f82818d6640 10 monclient: init 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient(hunting): picked mon.noname-a con 0x7f827c0a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:51.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient(hunting): picked mon.noname-b con 0x7f827c13f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:51.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient(hunting): picked mon.noname-c con 0x7f827c1430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:51.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c142520 con 0x7f827c0a98f0 2024-03-25T18:17:51.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0b28a0 con 0x7f827c1430c0 2024-03-25T18:17:51.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c0b2a30 con 0x7f827c13f710 2024-03-25T18:17:51.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:51.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.735+0000 7f82818d6640 10 monclient(hunting): authenticate will time out at 7511.998266s 2024-03-25T18:17:51.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.743+0000 7f827a7fc640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f827c13f710 0x7f827c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:44986/0 (socket says 172.21.15.132:44986) 2024-03-25T18:17:51.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.743+0000 7f827a7fc640 1 -- 172.21.15.132:0/3826521592 learned_addr learned my addr 172.21.15.132:0/3826521592 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:51.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.750+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006597000 0 0) 0x7f827c0b2a30 con 0x7f827c13f710 2024-03-25T18:17:51.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.750+0000 7f8279ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.750+0000 7f8279ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.750+0000 7f8279ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.750+0000 7f8279ffb640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:17:51.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.750+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8264003520 con 0x7f827c13f710 2024-03-25T18:17:51.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.756+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3537410680 0 0) 0x7f8264003520 con 0x7f827c13f710 2024-03-25T18:17:51.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.756+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 >> v1:172.21.15.161:6789/0 conn(0x7f827c1430c0 legacy=0x7f827c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.756+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 >> v1:172.21.15.132:6789/0 conn(0x7f827c0a98f0 legacy=0x7f827c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.758 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.756+0000 7f8279ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.756+0000 7f8279ffb640 1 monclient: found mon.noname-b 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.756+0000 7f8279ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0b44c0 con 0x7f827c13f710 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8270001690 con 0x7f827c13f710 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:51.759 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f8279ffb640 10 monclient: dump: 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:51.760 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f82818d6640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:17:51.761 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.757+0000 7f82818d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82700017f0 con 0x7f827c13f710 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f8279ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f82818d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f82818d6640 4 set_mon_vals no callback set 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f8279ffb640 1 -- 172.21.15.132:0/3826521592 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8270001c00 con 0x7f827c13f710 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f8279ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f8279ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.759+0000 7f8279ffb640 10 monclient: dump: 2024-03-25T18:17:51.764 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:51.765 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 10 monclient: shutdown 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 1 -- 172.21.15.132:0/3826521592 >> v1:172.21.15.132:6790/0 conn(0x7f827c13f710 legacy=0x7f827c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 1 -- 172.21.15.132:0/3826521592 shutdown_connections 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 1 -- 172.21.15.132:0/3826521592 >> 172.21.15.132:0/3826521592 conn(0x7f827c0b7ea0 msgr2=0x7f827c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 1 -- 172.21.15.132:0/3826521592 shutdown_connections 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.760+0000 7f82818d6640 1 -- 172.21.15.132:0/3826521592 wait complete. 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 10 monclient: build_initial_monmap 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 10 monclient: monmap: 2024-03-25T18:17:51.766 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 1 librados: starting msgr at 2024-03-25T18:17:51.767 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 1 librados: starting objecter 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 1 Processor -- start 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 1 -- start start 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 1 librados: setting wanted keys 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 1 librados: calling monclient init 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.761+0000 7f82818d6640 10 monclient: init 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient(hunting): picked mon.noname-a con 0x7f827c0a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient(hunting): picked mon.noname-b con 0x7f827c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient(hunting): picked mon.noname-c con 0x7f827c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:51.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c13aea0 con 0x7f827c0a98f0 2024-03-25T18:17:51.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c13b110 con 0x7f827c13f710 2024-03-25T18:17:51.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827c13b3f0 con 0x7f827c1430c0 2024-03-25T18:17:51.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:51.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.762+0000 7f82818d6640 10 monclient(hunting): authenticate will time out at 7512.025246s 2024-03-25T18:17:51.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.766+0000 7f827b7fe640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f827c1430c0 0x7f827c1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:45022/0 (socket says 172.21.15.132:45022) 2024-03-25T18:17:51.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.766+0000 7f827b7fe640 1 -- 172.21.15.132:0/2834090710 learned_addr learned my addr 172.21.15.132:0/2834090710 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:51.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.769+0000 7f8263fff640 10 client.?.objecter ms_handle_connect 0x7f827c1430c0 2024-03-25T18:17:51.770 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.769+0000 7f8263fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.776+0000 7f8263fff640 10 client.?.objecter ms_handle_connect 0x7f827c13f710 2024-03-25T18:17:51.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.776+0000 7f8263fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.782+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583812383 0 0) 0x7f827c13b3f0 con 0x7f827c1430c0 2024-03-25T18:17:51.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.782+0000 7f8263fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.782+0000 7f8263fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.784 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.782+0000 7f8263fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.784 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.782+0000 7f8263fff640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:17:51.784 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.782+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8268004eb0 con 0x7f827c1430c0 2024-03-25T18:17:51.785 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.784+0000 7f8263fff640 10 client.?.objecter ms_handle_connect 0x7f827c0a98f0 2024-03-25T18:17:51.785 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.784+0000 7f8263fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1304475238 0 0) 0x7f8268004eb0 con 0x7f827c1430c0 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f827c13b3f0 con 0x7f827c1430c0 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8274001c90 con 0x7f827c1430c0 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:51.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 10 monclient(hunting): dump: 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:51.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.789+0000 7f8263fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:51.793 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.792+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955672790 0 0) 0x7f827c13b110 con 0x7f827c13f710 2024-03-25T18:17:51.793 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.792+0000 7f8263fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.793 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.792+0000 7f8263fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.792+0000 7f8263fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.792+0000 7f8263fff640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:17:51.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.792+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8268004eb0 con 0x7f827c13f710 2024-03-25T18:17:51.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.796+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079472400 0 0) 0x7f827c13aea0 con 0x7f827c0a98f0 2024-03-25T18:17:51.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.796+0000 7f8263fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:51.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.796+0000 7f8263fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:51.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.796+0000 7f8263fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:51.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.796+0000 7f8263fff640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:17:51.798 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.796+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f827c13b110 con 0x7f827c0a98f0 2024-03-25T18:17:51.800 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2036595538 0 0) 0x7f827c13b3f0 con 0x7f827c1430c0 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 >> v1:172.21.15.161:6789/0 conn(0x7f827c13f710 legacy=0x7f827c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 >> v1:172.21.15.132:6789/0 conn(0x7f827c0a98f0 legacy=0x7f827c139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 1 monclient: found mon.c 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c0b71a0 con 0x7f827c1430c0 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f8263fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:51.801 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 5 monclient: authenticate success, global_id 4412 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 10 monclient: _renew_subs 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827c13a4c0 con 0x7f827c1430c0 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 10 monclient: _renew_subs 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f827c130700 con 0x7f827c1430c0 2024-03-25T18:17:51.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.799+0000 7f82818d6640 1 librados: init done 2024-03-25T18:17:51.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.800+0000 7f82818d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:51.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.800+0000 7f82818d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:51.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.800+0000 7f82818d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:17:51.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:51.800+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827c0627b0 con 0x7f827c1430c0 2024-03-25T18:17:52.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8274003990 con 0x7f827c1430c0 2024-03-25T18:17:52.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f8263fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:52.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8274003da0 con 0x7f827c1430c0 2024-03-25T18:17:52.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f8263fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f8263fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f8263fff640 10 monclient: dump: 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:52.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.105+0000 7f82797fa640 4 set_mon_vals no callback set 2024-03-25T18:17:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.106+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f8274003fc0 con 0x7f827c1430c0 2024-03-25T18:17:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.106+0000 7f827affd640 10 monclient: get_auth_request con 0x7f826806de60 auth_method 0 2024-03-25T18:17:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.107+0000 7f8263fff640 10 client.4412.objecter ms_handle_connect 0x7f826806de60 2024-03-25T18:17:52.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f8274054b50 con 0x7f827c1430c0 2024-03-25T18:17:52.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 10 client.4412.objecter ms_dispatch 0x7f827c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:52.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:52.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:52.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T18:17:52.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8274050660 con 0x7f827c1430c0 2024-03-25T18:17:52.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.113+0000 7f8263fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:52.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.344+0000 7f82818d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.344+0000 7f82818d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.344+0000 7f82818d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:17:52.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.344+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f827c0b7330 con 0x7f827c1430c0 2024-03-25T18:17:52.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.366+0000 7f8263fff640 1 -- 172.21.15.132:0/2834090710 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f8274052590 con 0x7f827c1430c0 2024-03-25T18:17:52.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.366+0000 7f8263fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:52.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.366+0000 7f8263fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:52.368 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:52.368 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:52.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 10 librados: watch_flush enter 2024-03-25T18:17:52.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 10 librados: watch_flush exit 2024-03-25T18:17:52.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T18:17:52.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 10 client.4412.objecter successfully canceled tick 2024-03-25T18:17:52.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 >> v1:172.21.15.161:6800/411950978 conn(0x7f826806de60 legacy=0x7f8268070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 10 monclient: shutdown 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 >> v1:172.21.15.132:6790/0 conn(0x7f827c1430c0 legacy=0x7f827c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 shutdown_connections 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.369+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 >> 172.21.15.132:0/2834090710 conn(0x7f827c0b7ea0 msgr2=0x7f827c13e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.370+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 shutdown_connections 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.370+0000 7f82818d6640 1 -- 172.21.15.132:0/2834090710 wait complete. 2024-03-25T18:17:52.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:52.370+0000 7f82818d6640 1 librados: shutdown 2024-03-25T18:17:52.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:54.670 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:04:39.934 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.721 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:04:39.985 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.750 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:04:40.014 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.771 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:04:40.013 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.776 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:04:40.040 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.781 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:04:40.023 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.783 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:04:40.047 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.805 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:04:40.069 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:54.815 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:04:40.056 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:59.188 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.282+0000 7f9f11ed3640 10 monclient: get_monmap_and_config 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.282+0000 7f9f11ed3640 10 monclient: build_initial_monmap 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.282+0000 7f9f11ed3640 10 monclient: monmap: 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:17:59.283357+0000 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:17:59.283357+0000 2024-03-25T18:17:59.283 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:59.284 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:59.284 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:59.284 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:17:59.284 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:17:59.284 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 1 Processor -- start 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 1 -- start start 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient: init 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f9f0c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f9f0c0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:59.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f9f0c14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:59.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c0af940 con 0x7f9f0c0b1730 2024-03-25T18:17:59.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c0b0a70 con 0x7f9f0c14be90 2024-03-25T18:17:59.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c0b0d50 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:59.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.283+0000 7f9f11ed3640 10 monclient(hunting): authenticate will time out at 7519.546465s 2024-03-25T18:17:59.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.308+0000 7f9f0b7fe640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f9f0c0b4d60 0x7f9f0c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:34128/0 (socket says 172.21.15.132:34128) 2024-03-25T18:17:59.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.308+0000 7f9f0b7fe640 1 -- 172.21.15.132:0/2947036073 learned_addr learned my addr 172.21.15.132:0/2947036073 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:59.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.325+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658389233 0 0) 0x7f9f0c0b0d50 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.325+0000 7f9f0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.325+0000 7f9f0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.325+0000 7f9f0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.325+0000 7f9f0a7fc640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:17:59.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.325+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef4003520 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.335+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1351048519 0 0) 0x7f9f0c0b0a70 con 0x7f9f0c14be90 2024-03-25T18:17:59.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.335+0000 7f9f0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.335+0000 7f9f0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.335+0000 7f9f0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.335+0000 7f9f0a7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:17:59.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.335+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c0b0d50 con 0x7f9f0c14be90 2024-03-25T18:17:59.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.344+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 21024299 0 0) 0x7f9f0c0b0d50 con 0x7f9f0c14be90 2024-03-25T18:17:59.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.344+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 >> v1:172.21.15.132:6790/0 conn(0x7f9f0c0b4d60 legacy=0x7f9f0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.344+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 >> v1:172.21.15.132:6789/0 conn(0x7f9f0c0b1730 legacy=0x7f9f0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:59.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:59.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c0b1030 con 0x7f9f0c14be90 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f040019f0 con 0x7f9f0c14be90 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f11ed3640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f11ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f0a7fc640 10 monclient: dump: 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.347 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:59.348 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:59.348 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:59.348 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:59.348 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.345+0000 7f9f11ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:59.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f04001c10 con 0x7f9f0c14be90 2024-03-25T18:17:59.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:59.367 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f0a7fc640 1 -- 172.21.15.132:0/2947036073 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f04001fa0 con 0x7f9f0c14be90 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f11ed3640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f11ed3640 4 set_mon_vals no callback set 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.366+0000 7f9f0a7fc640 10 monclient: dump: 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:59.368 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:59.369 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:59.369 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 10 monclient: shutdown 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 1 -- 172.21.15.132:0/2947036073 >> v1:172.21.15.161:6789/0 conn(0x7f9f0c14be90 legacy=0x7f9f0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 1 -- 172.21.15.132:0/2947036073 shutdown_connections 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 1 -- 172.21.15.132:0/2947036073 >> 172.21.15.132:0/2947036073 conn(0x7f9f0c0617a0 msgr2=0x7f9f0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 1 -- 172.21.15.132:0/2947036073 shutdown_connections 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.367+0000 7f9f11ed3640 1 -- 172.21.15.132:0/2947036073 wait complete. 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.368+0000 7f9f11ed3640 10 monclient: build_initial_monmap 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.368+0000 7f9f11ed3640 10 monclient: monmap: 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:17:59.371 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.368+0000 7f9f11ed3640 1 librados: starting msgr at 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.368+0000 7f9f11ed3640 1 librados: starting objecter 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.368+0000 7f9f11ed3640 1 Processor -- start 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.368+0000 7f9f11ed3640 1 -- start start 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 1 librados: setting wanted keys 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 1 librados: calling monclient init 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 10 monclient: init 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f9f0c0b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:17:59.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f9f0c0b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:17:59.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.369+0000 7f9f11ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f9f0c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:17:59.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.370+0000 7f9f11ed3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c141ca0 con 0x7f9f0c14be90 2024-03-25T18:17:59.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.370+0000 7f9f11ed3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c143420 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.370+0000 7f9f11ed3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f0c140810 con 0x7f9f0c0b1730 2024-03-25T18:17:59.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.370+0000 7f9f11ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:59.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.370+0000 7f9f11ed3640 10 monclient(hunting): authenticate will time out at 7519.632808s 2024-03-25T18:17:59.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.375+0000 7f9f0bfff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f9f0c14be90 0x7f9f0c1d7900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:44278/0 (socket says 172.21.15.132:44278) 2024-03-25T18:17:59.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.375+0000 7f9f0bfff640 1 -- 172.21.15.132:0/269446129 learned_addr learned my addr 172.21.15.132:0/269446129 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:17:59.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.375+0000 7f9ef3fff640 10 client.?.objecter ms_handle_connect 0x7f9f0c14be90 2024-03-25T18:17:59.376 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.375+0000 7f9ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.380+0000 7f9ef3fff640 10 client.?.objecter ms_handle_connect 0x7f9f0c0b4d60 2024-03-25T18:17:59.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.380+0000 7f9ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77077535 0 0) 0x7f9f0c141ca0 con 0x7f9f0c14be90 2024-03-25T18:17:59.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:17:59.384 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef8004eb0 con 0x7f9f0c14be90 2024-03-25T18:17:59.384 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 10 client.?.objecter ms_handle_connect 0x7f9f0c0b1730 2024-03-25T18:17:59.384 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.382+0000 7f9ef3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:59.385 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.384+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776313011 0 0) 0x7f9f0c143420 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.385 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.384+0000 7f9ef3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.385 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.384+0000 7f9ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.385 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.384+0000 7f9ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.385 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.384+0000 7f9ef3fff640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:17:59.386 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.384+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f0c141ca0 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.390+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1970954410 0 0) 0x7f9ef8004eb0 con 0x7f9f0c14be90 2024-03-25T18:17:59.391 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.390+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0c143420 con 0x7f9f0c14be90 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.391+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f04002020 con 0x7f9f0c14be90 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.391+0000 7f9ef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.391+0000 7f9ef3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.391+0000 7f9ef3fff640 10 monclient(hunting): dump: 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:59.392 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:59.393 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:59.393 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:59.393 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.391+0000 7f9ef3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:59.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.393+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731519269 0 0) 0x7f9f0c140810 con 0x7f9f0c0b1730 2024-03-25T18:17:59.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.393+0000 7f9ef3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:59.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.393+0000 7f9ef3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:59.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.393+0000 7f9ef3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:59.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.393+0000 7f9ef3fff640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:17:59.394 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.393+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef8004eb0 con 0x7f9f0c0b1730 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.399+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2712724619 0 0) 0x7f9f0c141ca0 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.399+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f0c140810 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.399+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f000018b0 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.399+0000 7f9ef3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.399+0000 7f9ef3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.399+0000 7f9ef3fff640 10 monclient(hunting): dump: 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:59.400 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:59.401 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4085299245 0 0) 0x7f9f0c140810 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 >> v1:172.21.15.132:6790/0 conn(0x7f9f0c0b1730 legacy=0x7f9f0c1412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 >> v1:172.21.15.132:6789/0 conn(0x7f9f0c14be90 legacy=0x7f9f0c1d7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:59.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 1 monclient: found mon.b 2024-03-25T18:17:59.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0c140380 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9ef3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 10 monclient: _renew_subs 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f0c1405d0 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 10 monclient: _renew_subs 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f0c133710 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.410+0000 7f9f11ed3640 1 librados: init done 2024-03-25T18:17:59.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.411+0000 7f9f11ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.411+0000 7f9f11ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.411+0000 7f9f11ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:59.413 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.411+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f0c141a00 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f00001d40 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9ef3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:59.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f00003650 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9ef3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:59.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9ef3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9ef3fff640 10 monclient: dump: 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:17:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.683+0000 7f9f09ffb640 4 set_mon_vals no callback set 2024-03-25T18:17:59.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.684+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f9f000196e0 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.684+0000 7f9f0b7fe640 10 monclient: get_auth_request con 0x7f9ef806df20 auth_method 0 2024-03-25T18:17:59.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.685+0000 7f9ef3fff640 10 client.4426.objecter ms_handle_connect 0x7f9ef806df20 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f9f000533c0 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 10 client.4426.objecter ms_dispatch 0x7f9f0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f0004ed00 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:59.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.691+0000 7f9ef3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:59.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.937+0000 7f9f11ed3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:59.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.937+0000 7f9f11ed3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:59.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.937+0000 7f9f11ed3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:17:59.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.937+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f0c1d7620 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.965+0000 7f9ef3fff640 1 -- 172.21.15.132:0/269446129 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f9f0c1d7620 con 0x7f9f0c0b4d60 2024-03-25T18:17:59.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.965+0000 7f9ef3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:59.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.965+0000 7f9ef3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:59.967 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:17:59.967 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:59.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 10 librados: watch_flush enter 2024-03-25T18:17:59.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 10 librados: watch_flush exit 2024-03-25T18:17:59.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T18:17:59.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 10 client.4426.objecter successfully canceled tick 2024-03-25T18:17:59.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 >> v1:172.21.15.161:6800/411950978 conn(0x7f9ef806df20 legacy=0x7f9ef80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 10 monclient: shutdown 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 >> v1:172.21.15.161:6789/0 conn(0x7f9f0c0b4d60 legacy=0x7f9f0c132e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 shutdown_connections 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.967+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 >> 172.21.15.132:0/269446129 conn(0x7f9f0c0617a0 msgr2=0x7f9f0c0b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.968+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 shutdown_connections 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.968+0000 7f9f11ed3640 1 -- 172.21.15.132:0/269446129 wait complete. 2024-03-25T18:17:59.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:17:59.968+0000 7f9f11ed3640 1 librados: shutdown 2024-03-25T18:17:59.980 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:06.786 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.898+0000 7fe17aae6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.898+0000 7fe17aae6640 10 monclient: build_initial_monmap 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.898+0000 7fe17aae6640 10 monclient: monmap: 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:06.899344+0000 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:06.899344+0000 2024-03-25T18:18:06.899 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:06.900 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:06.900 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:06.900 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:06.900 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:06.900 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:06.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 1 Processor -- start 2024-03-25T18:18:06.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 1 -- start start 2024-03-25T18:18:06.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 10 monclient: init 2024-03-25T18:18:06.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:06.901 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 10 monclient(hunting): picked mon.noname-c con 0x7fe1740b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.899+0000 7fe17aae6640 10 monclient(hunting): picked mon.noname-a con 0x7fe1740b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.900+0000 7fe17aae6640 10 monclient(hunting): picked mon.noname-b con 0x7fe17414be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.900+0000 7fe17aae6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1740af940 con 0x7fe1740b1730 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.900+0000 7fe17aae6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1740b0a70 con 0x7fe1740b4d60 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.900+0000 7fe17aae6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1740b0d50 con 0x7fe17414be90 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.900+0000 7fe17aae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:06.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.900+0000 7fe17aae6640 10 monclient(hunting): authenticate will time out at 7527.162911s 2024-03-25T18:18:06.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.909+0000 7fe173fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe1740b1730 0x7fe1740b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:56262/0 (socket says 172.21.15.132:56262) 2024-03-25T18:18:06.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.909+0000 7fe178882640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe1740b4d60 0x7fe17413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:53594/0 (socket says 172.21.15.132:53594) 2024-03-25T18:18:06.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.909+0000 7fe173fff640 1 -- 172.21.15.132:0/1687153611 learned_addr learned my addr 172.21.15.132:0/1687153611 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:06.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.917+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360757069 0 0) 0x7fe1740af940 con 0x7fe1740b1730 2024-03-25T18:18:06.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.917+0000 7fe1737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.917+0000 7fe1737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.917+0000 7fe1737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.917+0000 7fe1737fe640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:18:06.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.917+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164003520 con 0x7fe1740b1730 2024-03-25T18:18:06.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.920+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553310255 0 0) 0x7fe1740b0d50 con 0x7fe17414be90 2024-03-25T18:18:06.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.920+0000 7fe1737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.920+0000 7fe1737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.920+0000 7fe1737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.920+0000 7fe1737fe640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:18:06.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.920+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1740af940 con 0x7fe17414be90 2024-03-25T18:18:06.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.927+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359777551 0 0) 0x7fe1740b0a70 con 0x7fe1740b4d60 2024-03-25T18:18:06.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.927+0000 7fe1737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.927+0000 7fe1737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.927+0000 7fe1737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.927+0000 7fe1737fe640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:18:06.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.927+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1740b0d50 con 0x7fe1740b4d60 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296183536 0 0) 0x7fe1740af940 con 0x7fe17414be90 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 >> v1:172.21.15.161:6789/0 conn(0x7fe1740b4d60 legacy=0x7fe17413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 >> v1:172.21.15.132:6789/0 conn(0x7fe1740b1730 legacy=0x7fe1740b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 1 monclient: found mon.noname-b 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1740b1030 con 0x7fe17414be90 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe16c0019f0 con 0x7fe17414be90 2024-03-25T18:18:06.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe1737fe640 10 monclient: dump: 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe17aae6640 5 monclient: authenticate success, global_id 4421 2024-03-25T18:18:06.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.928+0000 7fe17aae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe16c001b50 con 0x7fe17414be90 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe1737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe1737fe640 1 -- 172.21.15.132:0/1687153611 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe16c001f60 con 0x7fe17414be90 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe17aae6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe17aae6640 4 set_mon_vals no callback set 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe1737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe1737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe1737fe640 10 monclient: dump: 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:06.938 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.939 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:06.939 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:06.939 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:06.939 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:06.939 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe17aae6640 10 monclient: shutdown 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe17aae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.937+0000 7fe17aae6640 1 -- 172.21.15.132:0/1687153611 >> v1:172.21.15.132:6790/0 conn(0x7fe17414be90 legacy=0x7fe17414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.938+0000 7fe17aae6640 1 -- 172.21.15.132:0/1687153611 shutdown_connections 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.938+0000 7fe17aae6640 1 -- 172.21.15.132:0/1687153611 >> 172.21.15.132:0/1687153611 conn(0x7fe1740617a0 msgr2=0x7fe174061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.938+0000 7fe17aae6640 1 -- 172.21.15.132:0/1687153611 shutdown_connections 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.938+0000 7fe17aae6640 1 -- 172.21.15.132:0/1687153611 wait complete. 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.938+0000 7fe17aae6640 10 monclient: build_initial_monmap 2024-03-25T18:18:06.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.938+0000 7fe17aae6640 10 monclient: monmap: 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 1 librados: starting msgr at 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 1 librados: starting objecter 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 1 Processor -- start 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 1 -- start start 2024-03-25T18:18:06.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 1 librados: setting wanted keys 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 1 librados: calling monclient init 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.939+0000 7fe17aae6640 10 monclient: init 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient(hunting): picked mon.noname-a con 0x7fe1740b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient(hunting): picked mon.noname-b con 0x7fe1740b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient(hunting): picked mon.noname-c con 0x7fe17414be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1741d77c0 con 0x7fe1740b1730 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1741d8360 con 0x7fe1740b4d60 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1741d84f0 con 0x7fe17414be90 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:06.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.940+0000 7fe17aae6640 10 monclient(hunting): authenticate will time out at 7527.202944s 2024-03-25T18:18:06.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.951+0000 7fe178882640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe1740b1730 0x7fe1741d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:56296/0 (socket says 172.21.15.132:56296) 2024-03-25T18:18:06.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.951+0000 7fe178882640 1 -- 172.21.15.132:0/1069637818 learned_addr learned my addr 172.21.15.132:0/1069637818 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:06.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.953+0000 7fe1717fa640 10 client.?.objecter ms_handle_connect 0x7fe1740b1730 2024-03-25T18:18:06.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.953+0000 7fe1717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:06.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.959+0000 7fe1717fa640 10 client.?.objecter ms_handle_connect 0x7fe1740b4d60 2024-03-25T18:18:06.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.959+0000 7fe1717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:06.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.964+0000 7fe1717fa640 10 client.?.objecter ms_handle_connect 0x7fe17414be90 2024-03-25T18:18:06.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.964+0000 7fe1717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:06.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.967+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663074732 0 0) 0x7fe1741d77c0 con 0x7fe1740b1730 2024-03-25T18:18:06.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.967+0000 7fe1717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.967+0000 7fe1717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.967+0000 7fe1717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.967+0000 7fe1717fa640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:18:06.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.967+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe158004eb0 con 0x7fe1740b1730 2024-03-25T18:18:06.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.973+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135508806 0 0) 0x7fe1741d8360 con 0x7fe1740b4d60 2024-03-25T18:18:06.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.973+0000 7fe1717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.973+0000 7fe1717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.973+0000 7fe1717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.973+0000 7fe1717fa640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:18:06.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.973+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1741d77c0 con 0x7fe1740b4d60 2024-03-25T18:18:06.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.984+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527118908 0 0) 0x7fe1741d84f0 con 0x7fe17414be90 2024-03-25T18:18:06.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.984+0000 7fe1717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:06.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.984+0000 7fe1717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:06.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.984+0000 7fe1717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:06.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.984+0000 7fe1717fa640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:18:06.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.984+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1741d8360 con 0x7fe17414be90 2024-03-25T18:18:06.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2660127370 0 0) 0x7fe1741d77c0 con 0x7fe1740b4d60 2024-03-25T18:18:06.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1741d84f0 con 0x7fe1740b4d60 2024-03-25T18:18:06.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe160001950 con 0x7fe1740b4d60 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 10 monclient(hunting): dump: 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:06.990 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:06.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:06.988+0000 7fe1717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.002+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713135083 0 0) 0x7fe158004eb0 con 0x7fe1740b1730 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.002+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1741d77c0 con 0x7fe1740b1730 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.002+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe1680019b0 con 0x7fe1740b1730 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.002+0000 7fe1717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.002+0000 7fe1717fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.002+0000 7fe1717fa640 10 monclient(hunting): dump: 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:07.003 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:07.004 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:07.004 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:07.004 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:07.011 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3119438166 0 0) 0x7fe1741d84f0 con 0x7fe1740b4d60 2024-03-25T18:18:07.011 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 >> v1:172.21.15.132:6790/0 conn(0x7fe17414be90 legacy=0x7fe174132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.011 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 >> v1:172.21.15.132:6789/0 conn(0x7fe1740b1730 legacy=0x7fe1741d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.011 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.011 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 1 monclient: found mon.b 2024-03-25T18:18:07.011 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe174140e40 con 0x7fe1740b4d60 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe1717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 10 monclient: _renew_subs 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe174141b30 con 0x7fe1740b4d60 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 10 monclient: _renew_subs 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe174133810 con 0x7fe1740b4d60 2024-03-25T18:18:07.012 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.010+0000 7fe17aae6640 1 librados: init done 2024-03-25T18:18:07.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.012+0000 7fe17aae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.012+0000 7fe17aae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.012+0000 7fe17aae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:07.013 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.012+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1740627b0 con 0x7fe1740b4d60 2024-03-25T18:18:07.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.266+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe160001d60 con 0x7fe1740b4d60 2024-03-25T18:18:07.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.266+0000 7fe1717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:07.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.266+0000 7fe172ffd640 4 set_mon_vals no callback set 2024-03-25T18:18:07.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.277+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe160004980 con 0x7fe1740b4d60 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.278+0000 7fe1717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.278+0000 7fe1717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.278+0000 7fe1717fa640 10 monclient: dump: 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:07.279 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.278+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fe160004b60 con 0x7fe1740b4d60 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.279+0000 7fe178882640 10 monclient: get_auth_request con 0x7fe15806df70 auth_method 0 2024-03-25T18:18:07.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.279+0000 7fe1717fa640 10 client.4432.objecter ms_handle_connect 0x7fe15806df70 2024-03-25T18:18:07.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fe160054780 con 0x7fe1740b4d60 2024-03-25T18:18:07.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 10 client.4432.objecter ms_dispatch 0x7fe174000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:07.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:07.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:07.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T18:18:07.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1600500c0 con 0x7fe1740b4d60 2024-03-25T18:18:07.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.283+0000 7fe1717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:07.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.526+0000 7fe17aae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.526+0000 7fe17aae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.526+0000 7fe17aae6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:07.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.526+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe174069f40 con 0x7fe1740b4d60 2024-03-25T18:18:07.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.566+0000 7fe1717fa640 1 -- 172.21.15.132:0/1069637818 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fe1600521c0 con 0x7fe1740b4d60 2024-03-25T18:18:07.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.566+0000 7fe1717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:07.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.566+0000 7fe1717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:07.567 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:07.567 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 10 librados: watch_flush enter 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 10 librados: watch_flush exit 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 10 client.4432.objecter successfully canceled tick 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 >> v1:172.21.15.161:6800/411950978 conn(0x7fe15806df70 legacy=0x7fe158070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 10 monclient: shutdown 2024-03-25T18:18:07.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:07.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 >> v1:172.21.15.161:6789/0 conn(0x7fe1740b4d60 legacy=0x7fe174141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 shutdown_connections 2024-03-25T18:18:07.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 >> 172.21.15.132:0/1069637818 conn(0x7fe1740617a0 msgr2=0x7fe17413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:07.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 shutdown_connections 2024-03-25T18:18:07.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.568+0000 7fe17aae6640 1 -- 172.21.15.132:0/1069637818 wait complete. 2024-03-25T18:18:07.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:07.569+0000 7fe17aae6640 1 librados: shutdown 2024-03-25T18:18:07.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:14.390 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.502+0000 7f6a54014640 10 monclient: get_monmap_and_config 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.502+0000 7f6a54014640 10 monclient: build_initial_monmap 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.502+0000 7f6a54014640 10 monclient: monmap: 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:14.503385+0000 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:14.503385+0000 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:14.503 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.504 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:14.504 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:14.504 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:14.504 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.502+0000 7f6a54014640 1 Processor -- start 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 1 -- start start 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient: init 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient(hunting): picked mon.noname-b con 0x7f6a4c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient(hunting): picked mon.noname-c con 0x7f6a4c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:14.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0af940 con 0x7f6a4c14be90 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0b0a70 con 0x7f6a4c0b1730 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0b0d50 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:14.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.503+0000 7f6a54014640 10 monclient(hunting): authenticate will time out at 7534.766677s 2024-03-25T18:18:14.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.508+0000 7f6a51db0640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f6a4c0b4d60 0x7f6a4c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:36246/0 (socket says 172.21.15.132:36246) 2024-03-25T18:18:14.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.509+0000 7f6a51db0640 1 -- 172.21.15.132:0/3179294352 learned_addr learned my addr 172.21.15.132:0/3179294352 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:14.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.513+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289054852 0 0) 0x7f6a4c0b0d50 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.513+0000 7f6a50dae640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.513+0000 7f6a50dae640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.513+0000 7f6a50dae640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.513+0000 7f6a50dae640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:18:14.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.513+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a38003520 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.519+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934719417 0 0) 0x7f6a38003520 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.519+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 >> v1:172.21.15.161:6789/0 conn(0x7f6a4c0b1730 legacy=0x7f6a4c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 >> v1:172.21.15.132:6789/0 conn(0x7f6a4c14be90 legacy=0x7f6a4c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: _finish_hunting 0 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 1 monclient: found mon.noname-b 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c0b1030 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: _finish_auth 0 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: _check_auth_tickets 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6a48001690 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a50dae640 10 monclient: dump: 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:14.522 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a54014640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:18:14.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.520+0000 7f6a54014640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:14.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a48001860 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a50dae640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:14.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a50dae640 1 -- 172.21.15.132:0/3179294352 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6a48001cb0 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a54014640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a54014640 4 set_mon_vals no callback set 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a50dae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a50dae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.531+0000 7f6a50dae640 10 monclient: dump: 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:14.533 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 10 monclient: shutdown 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 1 -- 172.21.15.132:0/3179294352 >> v1:172.21.15.132:6790/0 conn(0x7f6a4c0b4d60 legacy=0x7f6a4c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 1 -- 172.21.15.132:0/3179294352 shutdown_connections 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 1 -- 172.21.15.132:0/3179294352 >> 172.21.15.132:0/3179294352 conn(0x7f6a4c0617a0 msgr2=0x7f6a4c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 1 -- 172.21.15.132:0/3179294352 shutdown_connections 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.532+0000 7f6a54014640 1 -- 172.21.15.132:0/3179294352 wait complete. 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 10 monclient: build_initial_monmap 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 10 monclient: monmap: 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:14.535 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 1 librados: starting msgr at 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 1 librados: starting objecter 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 1 Processor -- start 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 1 -- start start 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 1 librados: setting wanted keys 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 1 librados: calling monclient init 2024-03-25T18:18:14.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.533+0000 7f6a54014640 10 monclient: init 2024-03-25T18:18:14.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:14.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:14.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient(hunting): picked mon.noname-b con 0x7f6a4c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:14.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4c0b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:14.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient(hunting): picked mon.noname-c con 0x7f6a4c14be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:14.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0aabb0 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c13b1b0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c138cd0 con 0x7f6a4c14be90 2024-03-25T18:18:14.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:14.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.534+0000 7f6a54014640 10 monclient(hunting): authenticate will time out at 7534.797608s 2024-03-25T18:18:14.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.540+0000 7f6a515af640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f6a4c0b4d60 0x7f6a4c137f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:45552/0 (socket says 172.21.15.132:45552) 2024-03-25T18:18:14.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.540+0000 7f6a515af640 1 -- 172.21.15.132:0/508633636 learned_addr learned my addr 172.21.15.132:0/508633636 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:14.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.540+0000 7f6a427fc640 10 client.?.objecter ms_handle_connect 0x7f6a4c0b4d60 2024-03-25T18:18:14.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.540+0000 7f6a427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.545+0000 7f6a427fc640 10 client.?.objecter ms_handle_connect 0x7f6a4c14be90 2024-03-25T18:18:14.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.545+0000 7f6a427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.548+0000 7f6a427fc640 10 client.?.objecter ms_handle_connect 0x7f6a4c0b1730 2024-03-25T18:18:14.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.548+0000 7f6a427fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:14.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.556+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 312520975 0 0) 0x7f6a4c0aabb0 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.556+0000 7f6a427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.556+0000 7f6a427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.556+0000 7f6a427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.556+0000 7f6a427fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.556+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a30004eb0 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.557+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159756915 0 0) 0x7f6a4c138cd0 con 0x7f6a4c14be90 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.557+0000 7f6a427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.557+0000 7f6a427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.557+0000 7f6a427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.557+0000 7f6a427fc640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:18:14.558 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.557+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c0aabb0 con 0x7f6a4c14be90 2024-03-25T18:18:14.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.561+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119687965 0 0) 0x7f6a4c13b1b0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.561+0000 7f6a427fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:14.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.561+0000 7f6a427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:14.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.561+0000 7f6a427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:14.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.561+0000 7f6a427fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:18:14.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.561+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c138cd0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2149273226 0 0) 0x7f6a4c138cd0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a4c13b1b0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6a48001b70 con 0x7f6a4c0b1730 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 10 monclient(hunting): dump: 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:14.579 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.578+0000 7f6a427fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:14.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.583+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4169262571 0 0) 0x7f6a4c0aabb0 con 0x7f6a4c14be90 2024-03-25T18:18:14.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.583+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a4c138cd0 con 0x7f6a4c14be90 2024-03-25T18:18:14.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.583+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6a44001a70 con 0x7f6a4c14be90 2024-03-25T18:18:14.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.583+0000 7f6a427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.583+0000 7f6a427fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.583+0000 7f6a427fc640 10 monclient(hunting): dump: 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:14.585 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.588+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248189298 0 0) 0x7f6a30004eb0 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.588+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a4c0aabb0 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.588+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6a3c001960 con 0x7f6a4c0b4d60 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.588+0000 7f6a427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.588+0000 7f6a427fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.588+0000 7f6a427fc640 10 monclient(hunting): dump: 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:14.590 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:14.591 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2615596323 0 0) 0x7f6a4c13b1b0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 >> v1:172.21.15.132:6790/0 conn(0x7f6a4c14be90 legacy=0x7f6a4c1e2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 >> v1:172.21.15.132:6789/0 conn(0x7f6a4c0b4d60 legacy=0x7f6a4c137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 1 monclient: found mon.b 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.603+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c138e60 con 0x7f6a4c0b1730 2024-03-25T18:18:14.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a427fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a427fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 10 monclient: _renew_subs 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c13be50 con 0x7f6a4c0b1730 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 10 monclient: _renew_subs 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c1db940 con 0x7f6a4c0b1730 2024-03-25T18:18:14.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.604+0000 7f6a54014640 1 librados: init done 2024-03-25T18:18:14.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.605+0000 7f6a54014640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.605+0000 7f6a54014640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.605+0000 7f6a54014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:14.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.605+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c1401f0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a48001cd0 con 0x7f6a4c0b1730 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6a48001250 con 0x7f6a4c0b1730 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a427fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a427fc640 10 monclient: dump: 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:14.888 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.886+0000 7f6a43fff640 4 set_mon_vals no callback set 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.887+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f6a4801aa60 con 0x7f6a4c0b1730 2024-03-25T18:18:14.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.888+0000 7f6a515af640 10 monclient: get_auth_request con 0x7f6a3006e0f0 auth_method 0 2024-03-25T18:18:14.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.888+0000 7f6a427fc640 10 client.4435.objecter ms_handle_connect 0x7f6a3006e0f0 2024-03-25T18:18:14.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f6a48054620 con 0x7f6a4c0b1730 2024-03-25T18:18:14.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 10 client.4435.objecter ms_dispatch 0x7f6a4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:14.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:14.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:14.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:18:14.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a48050130 con 0x7f6a4c0b1730 2024-03-25T18:18:14.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:14.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:14.893+0000 7f6a427fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:14.950 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:00.214 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.004 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:00.268 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.044 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:00.309 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.059 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:00.301 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.067 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:00.309 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.070 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:00.334 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.094 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:00.358 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.094+0000 7f6a54014640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.094+0000 7f6a54014640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.094+0000 7f6a54014640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:15.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.094+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a4c0b22f0 con 0x7f6a4c0b1730 2024-03-25T18:18:15.095 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:00.337 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.117 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:00.381 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:15.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.126+0000 7f6a427fc640 1 -- 172.21.15.132:0/508633636 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f6a4c0b22f0 con 0x7f6a4c0b1730 2024-03-25T18:18:15.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.126+0000 7f6a427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.126+0000 7f6a427fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:15.127 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:15.127 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:15.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 10 librados: watch_flush enter 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 10 librados: watch_flush exit 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 >> v1:172.21.15.161:6800/411950978 conn(0x7f6a3006e0f0 legacy=0x7f6a300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 10 monclient: shutdown 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 >> v1:172.21.15.161:6789/0 conn(0x7f6a4c0b1730 legacy=0x7f6a4c1db050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 shutdown_connections 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 >> 172.21.15.132:0/508633636 conn(0x7f6a4c0617a0 msgr2=0x7f6a4c144ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 shutdown_connections 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 -- 172.21.15.132:0/508633636 wait complete. 2024-03-25T18:18:15.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:15.128+0000 7f6a54014640 1 librados: shutdown 2024-03-25T18:18:15.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:21.946 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:22.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.060+0000 7f0594663640 10 monclient: get_monmap_and_config 2024-03-25T18:18:22.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.060+0000 7f0594663640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.060+0000 7f0594663640 10 monclient: monmap: 2024-03-25T18:18:22.061 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:22.061130+0000 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:22.061130+0000 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:22.062 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.061+0000 7f0594663640 1 Processor -- start 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.061+0000 7f0594663640 1 -- start start 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.061+0000 7f0594663640 10 monclient: init 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient(hunting): picked mon.noname-c con 0x7f058c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient(hunting): picked mon.noname-a con 0x7f058c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient(hunting): picked mon.noname-b con 0x7f058c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c142520 con 0x7f058c13f710 2024-03-25T18:18:22.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c0b28a0 con 0x7f058c0a98f0 2024-03-25T18:18:22.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c0b2a30 con 0x7f058c1430c0 2024-03-25T18:18:22.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.062+0000 7f0594663640 10 monclient(hunting): authenticate will time out at 7542.325453s 2024-03-25T18:18:22.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.078+0000 7f0592c00640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f058c1430c0 0x7f058c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:40602/0 (socket says 172.21.15.132:40602) 2024-03-25T18:18:22.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.078+0000 7f0592c00640 1 -- 172.21.15.132:0/2380899848 learned_addr learned my addr 172.21.15.132:0/2380899848 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:22.090 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.089+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318994329 0 0) 0x7f058c0b28a0 con 0x7f058c0a98f0 2024-03-25T18:18:22.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.089+0000 7f05913fd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.089+0000 7f05913fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.089+0000 7f05913fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.089+0000 7f05913fd640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:18:22.091 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.089+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0578003520 con 0x7f058c0a98f0 2024-03-25T18:18:22.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.091+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383358603 0 0) 0x7f058c142520 con 0x7f058c13f710 2024-03-25T18:18:22.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.091+0000 7f05913fd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.091+0000 7f05913fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.091+0000 7f05913fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.091+0000 7f05913fd640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:18:22.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.091+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c0b28a0 con 0x7f058c13f710 2024-03-25T18:18:22.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.097+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168772367 0 0) 0x7f058c0b2a30 con 0x7f058c1430c0 2024-03-25T18:18:22.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.097+0000 7f05913fd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.097+0000 7f05913fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.097+0000 7f05913fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.097+0000 7f05913fd640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:18:22.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.097+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c142520 con 0x7f058c1430c0 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155220682 0 0) 0x7f058c0b28a0 con 0x7f058c13f710 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 >> v1:172.21.15.132:6790/0 conn(0x7f058c1430c0 legacy=0x7f058c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 >> v1:172.21.15.161:6789/0 conn(0x7f058c0a98f0 legacy=0x7f058c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 1 monclient: found mon.noname-a 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0b44c0 con 0x7f058c13f710 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f057c001670 con 0x7f058c13f710 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f05913fd640 10 monclient: dump: 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:22.106 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f0594663640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:18:22.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.103+0000 7f0594663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f057c0017d0 con 0x7f058c13f710 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f05913fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f05913fd640 1 -- 172.21.15.132:0/2380899848 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f057c001be0 con 0x7f058c13f710 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f0594663640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f0594663640 4 set_mon_vals no callback set 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f05913fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f05913fd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f05913fd640 10 monclient: dump: 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:22.111 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:22.112 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:22.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f0594663640 10 monclient: shutdown 2024-03-25T18:18:22.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f0594663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:22.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.110+0000 7f0594663640 1 -- 172.21.15.132:0/2380899848 >> v1:172.21.15.132:6789/0 conn(0x7f058c13f710 legacy=0x7f058c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.111+0000 7f0594663640 1 -- 172.21.15.132:0/2380899848 shutdown_connections 2024-03-25T18:18:22.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.111+0000 7f0594663640 1 -- 172.21.15.132:0/2380899848 >> 172.21.15.132:0/2380899848 conn(0x7f058c0b7ea0 msgr2=0x7f058c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:22.114 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.111+0000 7f0594663640 1 -- 172.21.15.132:0/2380899848 shutdown_connections 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.111+0000 7f0594663640 1 -- 172.21.15.132:0/2380899848 wait complete. 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 10 monclient: monmap: 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:22.115 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 1 librados: starting msgr at 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 1 librados: starting objecter 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 1 Processor -- start 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 1 -- start start 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 1 librados: setting wanted keys 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 1 librados: calling monclient init 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 10 monclient: init 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.112+0000 7f0594663640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 10 monclient(hunting): picked mon.noname-b con 0x7f058c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 10 monclient(hunting): picked mon.noname-c con 0x7f058c13f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 10 monclient(hunting): picked mon.noname-a con 0x7f058c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c136750 con 0x7f058c1430c0 2024-03-25T18:18:22.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c1dac30 con 0x7f058c0a98f0 2024-03-25T18:18:22.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c1399c0 con 0x7f058c13f710 2024-03-25T18:18:22.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.113+0000 7f0594663640 10 monclient(hunting): authenticate will time out at 7542.375787s 2024-03-25T18:18:22.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.121+0000 7f05923ff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f058c0a98f0 0x7f058c1386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:33766/0 (socket says 172.21.15.132:33766) 2024-03-25T18:18:22.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.121+0000 7f05923ff640 1 -- 172.21.15.132:0/2638213099 learned_addr learned my addr 172.21.15.132:0/2638213099 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:22.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.126+0000 7f0582ffd640 10 client.?.objecter ms_handle_connect 0x7f058c0a98f0 2024-03-25T18:18:22.127 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.126+0000 7f0582ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.133+0000 7f0582ffd640 10 client.?.objecter ms_handle_connect 0x7f058c13f710 2024-03-25T18:18:22.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.133+0000 7f0582ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.135+0000 7f0582ffd640 10 client.?.objecter ms_handle_connect 0x7f058c1430c0 2024-03-25T18:18:22.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.135+0000 7f0582ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.138+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941323565 0 0) 0x7f058c1dac30 con 0x7f058c0a98f0 2024-03-25T18:18:22.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.138+0000 7f0582ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.138+0000 7f0582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.138+0000 7f0582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.138+0000 7f0582ffd640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:18:22.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.138+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0570004eb0 con 0x7f058c0a98f0 2024-03-25T18:18:22.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.144+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331047511 0 0) 0x7f058c1399c0 con 0x7f058c13f710 2024-03-25T18:18:22.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.144+0000 7f0582ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.144+0000 7f0582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.144+0000 7f0582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.145 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.144+0000 7f0582ffd640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:18:22.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.144+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f058c1dac30 con 0x7f058c13f710 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2303749380 0 0) 0x7f0570004eb0 con 0x7f058c0a98f0 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f058c1399c0 con 0x7f058c0a98f0 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0588001d10 con 0x7f058c0a98f0 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:22.153 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:22.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.152+0000 7f0582ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:22.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.158+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398934986 0 0) 0x7f058c136750 con 0x7f058c1430c0 2024-03-25T18:18:22.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.158+0000 7f0582ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.158+0000 7f0582ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.158+0000 7f0582ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.158+0000 7f0582ffd640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:18:22.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.158+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0570004eb0 con 0x7f058c1430c0 2024-03-25T18:18:22.163 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2074501531 0 0) 0x7f058c1399c0 con 0x7f058c0a98f0 2024-03-25T18:18:22.163 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 >> v1:172.21.15.132:6790/0 conn(0x7f058c13f710 legacy=0x7f058c1db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 >> v1:172.21.15.132:6789/0 conn(0x7f058c1430c0 legacy=0x7f058c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 1 monclient: found mon.b 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c1db860 con 0x7f058c0a98f0 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0582ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 10 monclient: _renew_subs 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c1d41e0 con 0x7f058c0a98f0 2024-03-25T18:18:22.164 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 10 monclient: _renew_subs 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c1d5150 con 0x7f058c0a98f0 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.162+0000 7f0594663640 1 librados: init done 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.163+0000 7f0594663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.163+0000 7f0594663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.163+0000 7f0594663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:22.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.163+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c1daa90 con 0x7f058c0a98f0 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0588003990 con 0x7f058c0a98f0 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0582ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0588003bc0 con 0x7f058c0a98f0 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0582ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0582ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0582ffd640 10 monclient: dump: 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.442 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.442+0000 7f0590bfc640 4 set_mon_vals no callback set 2024-03-25T18:18:22.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.443+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f0588003da0 con 0x7f058c0a98f0 2024-03-25T18:18:22.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.443+0000 7f0591bfe640 10 monclient: get_auth_request con 0x7f057006de00 auth_method 0 2024-03-25T18:18:22.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.443+0000 7f0582ffd640 10 client.4441.objecter ms_handle_connect 0x7f057006de00 2024-03-25T18:18:22.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.447+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f0588054a20 con 0x7f058c0a98f0 2024-03-25T18:18:22.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.447+0000 7f0582ffd640 10 client.4441.objecter ms_dispatch 0x7f058c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:22.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.447+0000 7f0582ffd640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:22.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.447+0000 7f0582ffd640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:22.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.447+0000 7f0582ffd640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:18:22.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.448+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0588050530 con 0x7f058c0a98f0 2024-03-25T18:18:22.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.448+0000 7f0582ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.448 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.448+0000 7f0582ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:22.673 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.673+0000 7f0594663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:22.673 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.673+0000 7f0594663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:22.673 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.673+0000 7f0594663640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:22.673 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.673+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f058c1dadc0 con 0x7f058c0a98f0 2024-03-25T18:18:22.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.699+0000 7f0582ffd640 1 -- 172.21.15.132:0/2638213099 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f0588052460 con 0x7f058c0a98f0 2024-03-25T18:18:22.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.699+0000 7f0582ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:22.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.699+0000 7f0582ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:22.699 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:22.699 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 10 librados: watch_flush enter 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 10 librados: watch_flush exit 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 >> v1:172.21.15.161:6800/411950978 conn(0x7f057006de00 legacy=0x7f05700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 10 monclient: shutdown 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 >> v1:172.21.15.161:6789/0 conn(0x7f058c0a98f0 legacy=0x7f058c1386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 shutdown_connections 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 >> 172.21.15.132:0/2638213099 conn(0x7f058c0b7ea0 msgr2=0x7f058c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 shutdown_connections 2024-03-25T18:18:22.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.701+0000 7f0594663640 1 -- 172.21.15.132:0/2638213099 wait complete. 2024-03-25T18:18:22.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:22.702+0000 7f0594663640 1 librados: shutdown 2024-03-25T18:18:22.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:29.520 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:29.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.633+0000 7f9f4bfe8640 10 monclient: get_monmap_and_config 2024-03-25T18:18:29.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.633+0000 7f9f4bfe8640 10 monclient: build_initial_monmap 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.633+0000 7f9f4bfe8640 10 monclient: monmap: 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:29.634881+0000 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:29.634881+0000 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:29.635 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:29.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.634+0000 7f9f4bfe8640 1 Processor -- start 2024-03-25T18:18:29.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.634+0000 7f9f4bfe8640 1 -- start start 2024-03-25T18:18:29.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.634+0000 7f9f4bfe8640 10 monclient: init 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f440b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f440b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f4414be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f440af940 con 0x7f9f4414be90 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f440b0a70 con 0x7f9f440b4d60 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f440b0d50 con 0x7f9f440b1730 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:29.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.635+0000 7f9f4bfe8640 10 monclient(hunting): authenticate will time out at 7549.898225s 2024-03-25T18:18:29.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.640+0000 7f9f49d84640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9f440b4d60 0x7f9f4413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:45652/0 (socket says 172.21.15.132:45652) 2024-03-25T18:18:29.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.640+0000 7f9f49d84640 1 -- 172.21.15.132:0/1002566691 learned_addr learned my addr 172.21.15.132:0/1002566691 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:29.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.657+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167482724 0 0) 0x7f9f440b0a70 con 0x7f9f440b4d60 2024-03-25T18:18:29.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.657+0000 7f9f48d82640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.657+0000 7f9f48d82640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.657+0000 7f9f48d82640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.657+0000 7f9f48d82640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:18:29.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.657+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f30003520 con 0x7f9f440b4d60 2024-03-25T18:18:29.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.660+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749338366 0 0) 0x7f9f440af940 con 0x7f9f4414be90 2024-03-25T18:18:29.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.660+0000 7f9f48d82640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.660+0000 7f9f48d82640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.660+0000 7f9f48d82640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.660+0000 7f9f48d82640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:18:29.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.660+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f440b0a70 con 0x7f9f4414be90 2024-03-25T18:18:29.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695132987 0 0) 0x7f9f440b0d50 con 0x7f9f440b1730 2024-03-25T18:18:29.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:18:29.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f440af940 con 0x7f9f440b1730 2024-03-25T18:18:29.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.669+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4224188449 0 0) 0x7f9f30003520 con 0x7f9f440b4d60 2024-03-25T18:18:29.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 >> v1:172.21.15.132:6790/0 conn(0x7f9f440b1730 legacy=0x7f9f440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 >> v1:172.21.15.132:6789/0 conn(0x7f9f4414be90 legacy=0x7f9f4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: _finish_hunting 0 2024-03-25T18:18:29.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 1 monclient: found mon.noname-c 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f440b1030 con 0x7f9f440b4d60 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: _finish_auth 0 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: _check_auth_tickets 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f40001690 con 0x7f9f440b4d60 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f4bfe8640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f4bfe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f48d82640 10 monclient: dump: 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:29.672 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:29.673 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.670+0000 7f9f4bfe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f400018b0 con 0x7f9f440b4d60 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f48d82640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f48d82640 1 -- 172.21.15.132:0/1002566691 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f40001c40 con 0x7f9f440b4d60 2024-03-25T18:18:29.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f48d82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f48d82640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f48d82640 10 monclient: dump: 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:29.686 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f4bfe8640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:29.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.684+0000 7f9f4bfe8640 4 set_mon_vals no callback set 2024-03-25T18:18:29.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 10 monclient: shutdown 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/1002566691 >> v1:172.21.15.161:6789/0 conn(0x7f9f440b4d60 legacy=0x7f9f4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/1002566691 shutdown_connections 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/1002566691 >> 172.21.15.132:0/1002566691 conn(0x7f9f440617a0 msgr2=0x7f9f44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/1002566691 shutdown_connections 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.685+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/1002566691 wait complete. 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 10 monclient: build_initial_monmap 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 10 monclient: monmap: 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:29.689 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 1 librados: starting msgr at 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 1 librados: starting objecter 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 1 Processor -- start 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 1 -- start start 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 1 librados: setting wanted keys 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 1 librados: calling monclient init 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.686+0000 7f9f4bfe8640 10 monclient: init 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient(hunting): picked mon.noname-a con 0x7f9f440b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:29.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient(hunting): picked mon.noname-b con 0x7f9f440b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:29.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient(hunting): picked mon.noname-c con 0x7f9f4414be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:29.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f44141f40 con 0x7f9f440b1730 2024-03-25T18:18:29.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f44142360 con 0x7f9f440b4d60 2024-03-25T18:18:29.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f441424f0 con 0x7f9f4414be90 2024-03-25T18:18:29.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:29.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.687+0000 7f9f4bfe8640 10 monclient(hunting): authenticate will time out at 7549.950463s 2024-03-25T18:18:29.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.699+0000 7f9f4a585640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f9f4414be90 0x7f9f441e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:42180/0 (socket says 172.21.15.132:42180) 2024-03-25T18:18:29.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.699+0000 7f9f4a585640 1 -- 172.21.15.132:0/3060358089 learned_addr learned my addr 172.21.15.132:0/3060358089 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.701+0000 7f9f3a7fc640 10 client.?.objecter ms_handle_connect 0x7f9f440b4d60 2024-03-25T18:18:29.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.701+0000 7f9f3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.702+0000 7f9f3a7fc640 10 client.?.objecter ms_handle_connect 0x7f9f4414be90 2024-03-25T18:18:29.703 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.702+0000 7f9f3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:29.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.707+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705547617 0 0) 0x7f9f440b0a70 con 0x7f9f4414be90 2024-03-25T18:18:29.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.707+0000 7f9f3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.707+0000 7f9f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.707+0000 7f9f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.707+0000 7f9f3a7fc640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:18:29.708 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.707+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f28004eb0 con 0x7f9f4414be90 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429120448 0 0) 0x7f9f44142360 con 0x7f9f440b4d60 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f440b0a70 con 0x7f9f440b4d60 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 10 client.?.objecter ms_handle_connect 0x7f9f440b1730 2024-03-25T18:18:29.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.713+0000 7f9f3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:29.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2591533830 0 0) 0x7f9f28004eb0 con 0x7f9f4414be90 2024-03-25T18:18:29.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f44142360 con 0x7f9f4414be90 2024-03-25T18:18:29.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f3c001af0 con 0x7f9f4414be90 2024-03-25T18:18:29.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 468625111 0 0) 0x7f9f440b0a70 con 0x7f9f440b4d60 2024-03-25T18:18:29.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f28004eb0 con 0x7f9f440b4d60 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f340018b0 con 0x7f9f440b4d60 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.727+0000 7f9f3a7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:29.730 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.730+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197161055 0 0) 0x7f9f44141f40 con 0x7f9f440b1730 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.730+0000 7f9f3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.730+0000 7f9f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.730+0000 7f9f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:29.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.730+0000 7f9f3a7fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:18:29.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.730+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f440b0a70 con 0x7f9f440b1730 2024-03-25T18:18:29.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3721707943 0 0) 0x7f9f44142360 con 0x7f9f4414be90 2024-03-25T18:18:29.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 >> v1:172.21.15.161:6789/0 conn(0x7f9f440b4d60 legacy=0x7f9f441d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 >> v1:172.21.15.132:6789/0 conn(0x7f9f440b1730 legacy=0x7f9f441405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:29.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:29.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 1 monclient: found mon.c 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f44140d00 con 0x7f9f4414be90 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 10 monclient: _renew_subs 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f44138150 con 0x7f9f4414be90 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 10 monclient: _renew_subs 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:29.739 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f441d8520 con 0x7f9f4414be90 2024-03-25T18:18:29.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.737+0000 7f9f4bfe8640 1 librados: init done 2024-03-25T18:18:29.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.738+0000 7f9f4bfe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:29.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.738+0000 7f9f4bfe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:29.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.738+0000 7f9f4bfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:29.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:29.738+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f44140f50 con 0x7f9f4414be90 2024-03-25T18:18:30.032 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f3c001fe0 con 0x7f9f4414be90 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f9f3c003b10 con 0x7f9f4414be90 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3a7fc640 10 monclient: dump: 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:30.033 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:30.034 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:30.034 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.031+0000 7f9f3bfff640 4 set_mon_vals no callback set 2024-03-25T18:18:30.034 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.032+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f9f3c003d30 con 0x7f9f4414be90 2024-03-25T18:18:30.034 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.033+0000 7f9f49d84640 10 monclient: get_auth_request con 0x7f9f2806df50 auth_method 0 2024-03-25T18:18:30.034 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.033+0000 7f9f3a7fc640 10 client.4442.objecter ms_handle_connect 0x7f9f2806df50 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f9f3c054890 con 0x7f9f4414be90 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 10 client.4442.objecter ms_dispatch 0x7f9f44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f3c0503a0 con 0x7f9f4414be90 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.038+0000 7f9f3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:30.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.286+0000 7f9f4bfe8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.286+0000 7f9f4bfe8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.286+0000 7f9f4bfe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:30.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.286+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f441d7950 con 0x7f9f4414be90 2024-03-25T18:18:30.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.323+0000 7f9f3a7fc640 1 -- 172.21.15.132:0/3060358089 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f9f441d7950 con 0x7f9f4414be90 2024-03-25T18:18:30.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.323+0000 7f9f3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:30.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.323+0000 7f9f3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:30.325 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:30.325 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.325+0000 7f9f4bfe8640 10 librados: watch_flush enter 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.325+0000 7f9f4bfe8640 10 librados: watch_flush exit 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.325+0000 7f9f4bfe8640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.325+0000 7f9f4bfe8640 10 client.4442.objecter successfully canceled tick 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.325+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 >> v1:172.21.15.161:6800/411950978 conn(0x7f9f2806df50 legacy=0x7f9f28070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.325+0000 7f9f4bfe8640 10 monclient: shutdown 2024-03-25T18:18:30.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:30.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 >> v1:172.21.15.132:6790/0 conn(0x7f9f4414be90 legacy=0x7f9f441e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 shutdown_connections 2024-03-25T18:18:30.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 >> 172.21.15.132:0/3060358089 conn(0x7f9f440617a0 msgr2=0x7f9f440b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:30.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 shutdown_connections 2024-03-25T18:18:30.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 1 -- 172.21.15.132:0/3060358089 wait complete. 2024-03-25T18:18:30.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:30.326+0000 7f9f4bfe8640 1 librados: shutdown 2024-03-25T18:18:30.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:34.817 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:20.081 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.826 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:20.090 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.852 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:20.116 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.871 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:20.112 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.872 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:20.136 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.879 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:20.143 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.879 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:20.120 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.920 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:20.161 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:34.926 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:20.190 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:37.146 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.261+0000 7efd2ecf1640 10 monclient: get_monmap_and_config 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.261+0000 7efd2ecf1640 10 monclient: build_initial_monmap 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.261+0000 7efd2ecf1640 10 monclient: monmap: 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:37.262892+0000 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:37.262892+0000 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:37.263 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.262+0000 7efd2ecf1640 1 Processor -- start 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.262+0000 7efd2ecf1640 1 -- start start 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.262+0000 7efd2ecf1640 10 monclient: init 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient(hunting): picked mon.noname-b con 0x7efd280a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient(hunting): picked mon.noname-a con 0x7efd2813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient(hunting): picked mon.noname-c con 0x7efd281430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd28142520 con 0x7efd2813f710 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280b28a0 con 0x7efd281430c0 2024-03-25T18:18:37.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280b2a30 con 0x7efd280a98f0 2024-03-25T18:18:37.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:37.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.263+0000 7efd2ecf1640 10 monclient(hunting): authenticate will time out at 7557.526075s 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.273+0000 7efd2d28e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efd281430c0 0x7efd2814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:43510/0 (socket says 172.21.15.132:43510) 2024-03-25T18:18:37.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.273+0000 7efd2d28e640 1 -- 172.21.15.132:0/2892219086 learned_addr learned my addr 172.21.15.132:0/2892219086 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:37.289 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.288+0000 7efd2ca8d640 1 -- 172.21.15.132:0/2892219086 >> v1:172.21.15.132:6790/0 conn(0x7efd280a98f0 legacy=0x7efd280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:18:37.289 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.288+0000 7efd2ca8d640 1 -- 172.21.15.132:0/2892219086 >> v1:172.21.15.132:6790/0 conn(0x7efd280a98f0 legacy=0x7efd280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:18:37.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.288+0000 7efd2ca8d640 1 --1- 172.21.15.132:0/2892219086 >> v1:172.21.15.132:6790/0 conn(0x7efd280a98f0 0x7efd280a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-25T18:18:37.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.298+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374322297 0 0) 0x7efd280b28a0 con 0x7efd281430c0 2024-03-25T18:18:37.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:18:37.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd18003520 con 0x7efd281430c0 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659169358 0 0) 0x7efd28142520 con 0x7efd2813f710 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:18:37.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.299+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd280b28a0 con 0x7efd2813f710 2024-03-25T18:18:37.305 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275694383 0 0) 0x7efd280b28a0 con 0x7efd2813f710 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 >> v1:172.21.15.132:6790/0 conn(0x7efd280a98f0 legacy=0x7efd280a9cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 >> v1:172.21.15.161:6789/0 conn(0x7efd281430c0 legacy=0x7efd2814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 1 monclient: found mon.noname-a 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd280b44c0 con 0x7efd2813f710 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd14001670 con 0x7efd2813f710 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd2ecf1640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd2ecf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:37.306 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd277fe640 10 monclient: dump: 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.304+0000 7efd2ecf1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:37.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd140017d0 con 0x7efd2813f710 2024-03-25T18:18:37.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:37.313 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd277fe640 1 -- 172.21.15.132:0/2892219086 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd14001c20 con 0x7efd2813f710 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd277fe640 10 monclient: dump: 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:37.314 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd2ecf1640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:37.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.312+0000 7efd2ecf1640 4 set_mon_vals no callback set 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.313+0000 7efd2ecf1640 10 monclient: shutdown 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.313+0000 7efd2ecf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.313+0000 7efd2ecf1640 1 -- 172.21.15.132:0/2892219086 >> v1:172.21.15.132:6789/0 conn(0x7efd2813f710 legacy=0x7efd281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.313+0000 7efd2ecf1640 1 -- 172.21.15.132:0/2892219086 shutdown_connections 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.313+0000 7efd2ecf1640 1 -- 172.21.15.132:0/2892219086 >> 172.21.15.132:0/2892219086 conn(0x7efd280b7ea0 msgr2=0x7efd280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.313+0000 7efd2ecf1640 1 -- 172.21.15.132:0/2892219086 shutdown_connections 2024-03-25T18:18:37.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.314+0000 7efd2ecf1640 1 -- 172.21.15.132:0/2892219086 wait complete. 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.314+0000 7efd2ecf1640 10 monclient: build_initial_monmap 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.314+0000 7efd2ecf1640 10 monclient: monmap: 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 1 librados: starting msgr at 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 1 librados: starting objecter 2024-03-25T18:18:37.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 1 Processor -- start 2024-03-25T18:18:37.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 1 -- start start 2024-03-25T18:18:37.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 1 librados: setting wanted keys 2024-03-25T18:18:37.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 1 librados: calling monclient init 2024-03-25T18:18:37.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.315+0000 7efd2ecf1640 10 monclient: init 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient(hunting): picked mon.noname-c con 0x7efd280a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient(hunting): picked mon.noname-a con 0x7efd2813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient(hunting): picked mon.noname-b con 0x7efd281430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280b1730 con 0x7efd2813f710 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280af0d0 con 0x7efd281430c0 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd280af3b0 con 0x7efd280a98f0 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:37.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.316+0000 7efd2ecf1640 10 monclient(hunting): authenticate will time out at 7557.578844s 2024-03-25T18:18:37.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.324+0000 7efd27fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7efd2813f710 0x7efd280acd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:48058/0 (socket says 172.21.15.132:48058) 2024-03-25T18:18:37.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.324+0000 7efd27fff640 1 -- 172.21.15.132:0/528215177 learned_addr learned my addr 172.21.15.132:0/528215177 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:37.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.325+0000 7efd257fa640 10 client.?.objecter ms_handle_connect 0x7efd2813f710 2024-03-25T18:18:37.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.325+0000 7efd257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.337+0000 7efd257fa640 10 client.?.objecter ms_handle_connect 0x7efd281430c0 2024-03-25T18:18:37.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.337+0000 7efd257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.345+0000 7efd257fa640 10 client.?.objecter ms_handle_connect 0x7efd280a98f0 2024-03-25T18:18:37.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.345+0000 7efd257fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.350+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702283891 0 0) 0x7efd280af0d0 con 0x7efd281430c0 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.350+0000 7efd257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.350+0000 7efd257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.350+0000 7efd257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.350+0000 7efd257fa640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:18:37.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.350+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd0c004eb0 con 0x7efd281430c0 2024-03-25T18:18:37.355 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677310202 0 0) 0x7efd280b44c0 con 0x7efd2813f710 2024-03-25T18:18:37.355 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd280af0d0 con 0x7efd2813f710 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443869543 0 0) 0x7efd280af3b0 con 0x7efd280a98f0 2024-03-25T18:18:37.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:37.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:37.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:37.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:18:37.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.355+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd280b44c0 con 0x7efd280a98f0 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 927916164 0 0) 0x7efd0c004eb0 con 0x7efd281430c0 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd280af3b0 con 0x7efd281430c0 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd20001b90 con 0x7efd281430c0 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 10 monclient(hunting): dump: 2024-03-25T18:18:37.364 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.363+0000 7efd257fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:37.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.373+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4126272755 0 0) 0x7efd280b44c0 con 0x7efd280a98f0 2024-03-25T18:18:37.374 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.373+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd0c004eb0 con 0x7efd280a98f0 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.373+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd1c001810 con 0x7efd280a98f0 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.373+0000 7efd257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.373+0000 7efd257fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.373+0000 7efd257fa640 10 monclient(hunting): dump: 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:37.375 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:37.376 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1770915362 0 0) 0x7efd280af3b0 con 0x7efd281430c0 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 >> v1:172.21.15.132:6790/0 conn(0x7efd280a98f0 legacy=0x7efd280b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 >> v1:172.21.15.132:6789/0 conn(0x7efd2813f710 legacy=0x7efd280acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 1 monclient: found mon.b 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd280b0990 con 0x7efd281430c0 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 5 monclient: authenticate success, global_id 4453 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 10 monclient: _renew_subs 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:37.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd280b0c00 con 0x7efd281430c0 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 10 monclient: _renew_subs 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd280af840 con 0x7efd281430c0 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.380+0000 7efd2ecf1640 1 librados: init done 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.381+0000 7efd2ecf1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.381+0000 7efd2ecf1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.381+0000 7efd2ecf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:37.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.381+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd280afb40 con 0x7efd281430c0 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd20001fe0 con 0x7efd281430c0 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7efd20004d00 con 0x7efd281430c0 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd257fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd257fa640 10 monclient: dump: 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:37.675 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:37.676 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.674+0000 7efd26ffd640 4 set_mon_vals no callback set 2024-03-25T18:18:37.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.675+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7efd20004f20 con 0x7efd281430c0 2024-03-25T18:18:37.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.675+0000 7efd2ca8d640 10 monclient: get_auth_request con 0x7efd0c06df20 auth_method 0 2024-03-25T18:18:37.677 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.676+0000 7efd257fa640 10 client.4453.objecter ms_handle_connect 0x7efd0c06df20 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.679+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7efd20054a40 con 0x7efd281430c0 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.679+0000 7efd257fa640 10 client.4453.objecter ms_dispatch 0x7efd28000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.679+0000 7efd257fa640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.679+0000 7efd257fa640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.679+0000 7efd257fa640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.680+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd20050550 con 0x7efd281430c0 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.681+0000 7efd257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:37.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.681+0000 7efd257fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:37.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.938+0000 7efd2ecf1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:37.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.938+0000 7efd2ecf1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:37.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.938+0000 7efd2ecf1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:37.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.938+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd280af540 con 0x7efd281430c0 2024-03-25T18:18:37.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.971+0000 7efd257fa640 1 -- 172.21.15.132:0/528215177 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7efd20052480 con 0x7efd281430c0 2024-03-25T18:18:37.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.971+0000 7efd257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:37.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.971+0000 7efd257fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:37.972 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:37.973 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.973+0000 7efd2ecf1640 10 librados: watch_flush enter 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.973+0000 7efd2ecf1640 10 librados: watch_flush exit 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.973+0000 7efd2ecf1640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.973+0000 7efd2ecf1640 10 client.4453.objecter successfully canceled tick 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 >> v1:172.21.15.161:6800/411950978 conn(0x7efd0c06df20 legacy=0x7efd0c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 10 monclient: shutdown 2024-03-25T18:18:37.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:37.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 >> v1:172.21.15.161:6789/0 conn(0x7efd281430c0 legacy=0x7efd280abd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:37.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 shutdown_connections 2024-03-25T18:18:37.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 >> 172.21.15.132:0/528215177 conn(0x7efd280b7ea0 msgr2=0x7efd28074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:37.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 shutdown_connections 2024-03-25T18:18:37.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 -- 172.21.15.132:0/528215177 wait complete. 2024-03-25T18:18:37.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:37.974+0000 7efd2ecf1640 1 librados: shutdown 2024-03-25T18:18:37.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:44.793 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:44.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.909+0000 7f3d6214b640 10 monclient: get_monmap_and_config 2024-03-25T18:18:44.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.909+0000 7f3d6214b640 10 monclient: build_initial_monmap 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.909+0000 7f3d6214b640 10 monclient: monmap: 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:44.910920+0000 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:44.910920+0000 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:44.911 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:44.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.910+0000 7f3d6214b640 1 Processor -- start 2024-03-25T18:18:44.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.910+0000 7f3d6214b640 1 -- start start 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.910+0000 7f3d6214b640 10 monclient: init 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c0b4f40 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c13fd90 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c0b4540 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c0aa0c0 con 0x7f3d5c0b4540 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c0ab170 con 0x7f3d5c13fd90 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c0a9ba0 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:44.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.911+0000 7f3d6214b640 10 monclient(hunting): authenticate will time out at 7565.174423s 2024-03-25T18:18:44.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.918+0000 7f3d5affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3d5c13fd90 0x7f3d5c1423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:33792/0 (socket says 172.21.15.132:33792) 2024-03-25T18:18:44.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.918+0000 7f3d5affd640 1 -- 172.21.15.132:0/2887996430 learned_addr learned my addr 172.21.15.132:0/2887996430 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:44.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.927+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1859588720 0 0) 0x7f3d5c0ab170 con 0x7f3d5c13fd90 2024-03-25T18:18:44.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.927+0000 7f3d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.927+0000 7f3d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.927+0000 7f3d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.927+0000 7f3d5a7fc640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:18:44.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.927+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d44003520 con 0x7f3d5c13fd90 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3659055757 0 0) 0x7f3d44003520 con 0x7f3d5c13fd90 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 >> v1:172.21.15.132:6790/0 conn(0x7f3d5c0b4f40 legacy=0x7f3d5c13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 >> v1:172.21.15.132:6789/0 conn(0x7f3d5c0b4540 legacy=0x7f3d5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c0a9680 con 0x7f3d5c13fd90 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:44.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3d50001690 con 0x7f3d5c13fd90 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d5a7fc640 10 monclient: dump: 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:44.935 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d6214b640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:18:44.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.932+0000 7f3d6214b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d500017f0 con 0x7f3d5c13fd90 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d5a7fc640 1 -- 172.21.15.132:0/2887996430 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3d50001c00 con 0x7f3d5c13fd90 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d5a7fc640 10 monclient: dump: 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:44.944 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d6214b640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:44.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.943+0000 7f3d6214b640 4 set_mon_vals no callback set 2024-03-25T18:18:44.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 10 monclient: shutdown 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 1 -- 172.21.15.132:0/2887996430 >> v1:172.21.15.161:6789/0 conn(0x7f3d5c13fd90 legacy=0x7f3d5c1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 1 -- 172.21.15.132:0/2887996430 shutdown_connections 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 1 -- 172.21.15.132:0/2887996430 >> 172.21.15.132:0/2887996430 conn(0x7f3d5c0617a0 msgr2=0x7f3d5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 1 -- 172.21.15.132:0/2887996430 shutdown_connections 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.944+0000 7f3d6214b640 1 -- 172.21.15.132:0/2887996430 wait complete. 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 10 monclient: build_initial_monmap 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 10 monclient: monmap: 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:44.947 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 1 librados: starting msgr at 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 1 librados: starting objecter 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 1 Processor -- start 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 1 -- start start 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 1 librados: setting wanted keys 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 1 librados: calling monclient init 2024-03-25T18:18:44.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.945+0000 7f3d6214b640 10 monclient: init 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5c0b4540 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5c0b4f40 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d5c13fd90 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c142430 con 0x7f3d5c0b4540 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c13e4f0 con 0x7f3d5c13fd90 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c141f30 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:44.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.946+0000 7f3d6214b640 10 monclient(hunting): authenticate will time out at 7565.209399s 2024-03-25T18:18:44.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.956+0000 7f3d5bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3d5c13fd90 0x7f3d5c1e3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:33832/0 (socket says 172.21.15.132:33832) 2024-03-25T18:18:44.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.956+0000 7f3d5bfff640 1 -- 172.21.15.132:0/3075631462 learned_addr learned my addr 172.21.15.132:0/3075631462 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:44.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.957+0000 7f3d43fff640 10 client.?.objecter ms_handle_connect 0x7f3d5c13fd90 2024-03-25T18:18:44.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.957+0000 7f3d43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:44.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.960+0000 7f3d43fff640 10 client.?.objecter ms_handle_connect 0x7f3d5c0b4540 2024-03-25T18:18:44.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.960+0000 7f3d43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:44.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.960+0000 7f3d43fff640 10 client.?.objecter ms_handle_connect 0x7f3d5c0b4f40 2024-03-25T18:18:44.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.960+0000 7f3d43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:44.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.967+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548267077 0 0) 0x7f3d5c142430 con 0x7f3d5c0b4540 2024-03-25T18:18:44.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.968+0000 7f3d43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.968+0000 7f3d43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.968+0000 7f3d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.968+0000 7f3d43fff640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:18:44.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.968+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d48004eb0 con 0x7f3d5c0b4540 2024-03-25T18:18:44.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.970+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633938574 0 0) 0x7f3d5c13e4f0 con 0x7f3d5c13fd90 2024-03-25T18:18:44.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.970+0000 7f3d43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.970+0000 7f3d43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.970+0000 7f3d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.970+0000 7f3d43fff640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:18:44.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.970+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c142430 con 0x7f3d5c13fd90 2024-03-25T18:18:44.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.973+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340746407 0 0) 0x7f3d5c141f30 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.973+0000 7f3d43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:44.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.973+0000 7f3d43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:44.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.973+0000 7f3d43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:44.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.973+0000 7f3d43fff640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:18:44.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.973+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d5c13e4f0 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.981+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1889444010 0 0) 0x7f3d48004eb0 con 0x7f3d5c0b4540 2024-03-25T18:18:44.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.981+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d5c141f30 con 0x7f3d5c0b4540 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.982+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3d4c001980 con 0x7f3d5c0b4540 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.982+0000 7f3d43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.982+0000 7f3d43fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.982+0000 7f3d43fff640 10 monclient(hunting): dump: 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:44.984 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:44.985 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:44.985 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:44.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.982+0000 7f3d43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.990+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1888661722 0 0) 0x7f3d5c142430 con 0x7f3d5c13fd90 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.990+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d48004eb0 con 0x7f3d5c13fd90 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.990+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3d54001c50 con 0x7f3d5c13fd90 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.990+0000 7f3d43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.990+0000 7f3d43fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.990+0000 7f3d43fff640 10 monclient(hunting): dump: 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:44.991 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:44.992 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:44.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.996+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668112879 0 0) 0x7f3d5c13e4f0 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.996+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d5c142430 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.996+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3d50001b00 con 0x7f3d5c0b4f40 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.996+0000 7f3d43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.996+0000 7f3d43fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:44.996+0000 7f3d43fff640 10 monclient(hunting): dump: 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:44.998 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:44.999 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:44.999 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:44.999 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3516538638 0 0) 0x7f3d5c142430 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 >> v1:172.21.15.161:6789/0 conn(0x7f3d5c13fd90 legacy=0x7f3d5c1e3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 >> v1:172.21.15.132:6789/0 conn(0x7f3d5c0b4540 legacy=0x7f3d5c1d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 1 monclient: found mon.c 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c1420c0 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.002+0000 7f3d43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 10 monclient: _renew_subs 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c139700 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 10 monclient: _renew_subs 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c1c9630 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.003+0000 7f3d6214b640 1 librados: init done 2024-03-25T18:18:45.006 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.004+0000 7f3d6214b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.006 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.004+0000 7f3d6214b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.006 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.004+0000 7f3d6214b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:45.006 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.004+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c139a50 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.281+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d50001d00 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.281+0000 7f3d43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:45.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.281+0000 7f3d59ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.293+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3d50004c30 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.293+0000 7f3d43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.293+0000 7f3d43fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.293+0000 7f3d43fff640 10 monclient: dump: 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:45.294 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:45.295 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.294+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f3d5001ac00 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.294+0000 7f3d5b7fe640 10 monclient: get_auth_request con 0x7f3d4806e0c0 auth_method 0 2024-03-25T18:18:45.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.295+0000 7f3d43fff640 10 client.4451.objecter ms_handle_connect 0x7f3d4806e0c0 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f3d50054790 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 10 client.4451.objecter ms_dispatch 0x7f3d5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d500500d0 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.300+0000 7f3d43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:45.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.503+0000 7f3d6214b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:45.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.503+0000 7f3d6214b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:45.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.503+0000 7f3d6214b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:18:45.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.503+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d5c0627b0 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.534+0000 7f3d43fff640 1 -- 172.21.15.132:0/3075631462 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f3d500521d0 con 0x7f3d5c0b4f40 2024-03-25T18:18:45.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.534+0000 7f3d43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:45.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.534+0000 7f3d43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:45.536 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:45.536 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:45.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 10 librados: watch_flush enter 2024-03-25T18:18:45.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 10 librados: watch_flush exit 2024-03-25T18:18:45.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:18:45.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:18:45.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 >> v1:172.21.15.161:6800/411950978 conn(0x7f3d4806e0c0 legacy=0x7f3d48070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 10 monclient: shutdown 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 >> v1:172.21.15.132:6790/0 conn(0x7f3d5c0b4f40 legacy=0x7f3d5c138ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 shutdown_connections 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.537+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 >> 172.21.15.132:0/3075631462 conn(0x7f3d5c0617a0 msgr2=0x7f3d5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.538+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 shutdown_connections 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.538+0000 7f3d6214b640 1 -- 172.21.15.132:0/3075631462 wait complete. 2024-03-25T18:18:45.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:45.538+0000 7f3d6214b640 1 librados: shutdown 2024-03-25T18:18:45.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:52.356 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.471+0000 7f4afa8c1640 10 monclient: get_monmap_and_config 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.471+0000 7f4afa8c1640 10 monclient: build_initial_monmap 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.471+0000 7f4afa8c1640 10 monclient: monmap: 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:18:52.472354+0000 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:18:52.472354+0000 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:18:52.472 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:18:52.473 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.472+0000 7f4afa8c1640 1 Processor -- start 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.472+0000 7f4afa8c1640 1 -- start start 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.472+0000 7f4afa8c1640 10 monclient: init 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4af40705b0 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4af41402d0 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4af4149ae0 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:52.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af40718c0 con 0x7f4af4149ae0 2024-03-25T18:18:52.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af40714c0 con 0x7f4af40705b0 2024-03-25T18:18:52.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af4072b50 con 0x7f4af41402d0 2024-03-25T18:18:52.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:52.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.473+0000 7f4afa8c1640 10 monclient(hunting): authenticate will time out at 7572.735867s 2024-03-25T18:18:52.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.478+0000 7f4af8e5e640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f4af4149ae0 0x7f4af414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:36392/0 (socket says 172.21.15.132:36392) 2024-03-25T18:18:52.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.478+0000 7f4af8e5e640 1 -- 172.21.15.132:0/457274525 learned_addr learned my addr 172.21.15.132:0/457274525 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:52.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.492+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351656855 0 0) 0x7f4af40718c0 con 0x7f4af4149ae0 2024-03-25T18:18:52.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.492+0000 7f4af2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.492+0000 7f4af2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.492+0000 7f4af2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.492+0000 7f4af2ffd640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:18:52.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.492+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4adc003520 con 0x7f4af4149ae0 2024-03-25T18:18:52.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417378863 0 0) 0x7f4af40714c0 con 0x7f4af40705b0 2024-03-25T18:18:52.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af40718c0 con 0x7f4af40705b0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1129172575 0 0) 0x7f4adc003520 con 0x7f4af4149ae0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 >> v1:172.21.15.132:6790/0 conn(0x7f4af41402d0 legacy=0x7f4af4140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 >> v1:172.21.15.161:6789/0 conn(0x7f4af40705b0 legacy=0x7f4af413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 monclient: found mon.noname-a 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af413dd10 con 0x7f4af4149ae0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4aec001a10 con 0x7f4af4149ae0 2024-03-25T18:18:52.507 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4afa8c1640 5 monclient: authenticate success, global_id 4464 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4afa8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4af2ffd640 10 monclient: dump: 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.508 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.504+0000 7f4afa8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aec001b70 con 0x7f4af4149ae0 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4af2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4af2ffd640 1 -- 172.21.15.132:0/457274525 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4aec001f80 con 0x7f4af4149ae0 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4afa8c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4afa8c1640 4 set_mon_vals no callback set 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4af2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4af2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4af2ffd640 10 monclient: dump: 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.521 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.522 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.522 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.522 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:52.522 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:52.522 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:52.522 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4afa8c1640 10 monclient: shutdown 2024-03-25T18:18:52.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4afa8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:52.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.520+0000 7f4afa8c1640 1 -- 172.21.15.132:0/457274525 >> v1:172.21.15.132:6789/0 conn(0x7f4af4149ae0 legacy=0x7f4af414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.523 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.521+0000 7f4afa8c1640 1 -- 172.21.15.132:0/457274525 shutdown_connections 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.521+0000 7f4afa8c1640 1 -- 172.21.15.132:0/457274525 >> 172.21.15.132:0/457274525 conn(0x7f4af40b5d10 msgr2=0x7f4af40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.521+0000 7f4afa8c1640 1 -- 172.21.15.132:0/457274525 shutdown_connections 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.521+0000 7f4afa8c1640 1 -- 172.21.15.132:0/457274525 wait complete. 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.521+0000 7f4afa8c1640 10 monclient: build_initial_monmap 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.521+0000 7f4afa8c1640 10 monclient: monmap: 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 1 librados: starting msgr at 2024-03-25T18:18:52.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 1 librados: starting objecter 2024-03-25T18:18:52.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 1 Processor -- start 2024-03-25T18:18:52.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 1 -- start start 2024-03-25T18:18:52.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 1 librados: setting wanted keys 2024-03-25T18:18:52.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 1 librados: calling monclient init 2024-03-25T18:18:52.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.522+0000 7f4afa8c1640 10 monclient: init 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4af40705b0 addr v1:172.21.15.132:6790/0 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4af41402d0 addr v1:172.21.15.132:6789/0 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4af4149ae0 addr v1:172.21.15.161:6789/0 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af406d800 con 0x7f4af41402d0 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af406dae0 con 0x7f4af4149ae0 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af406c440 con 0x7f4af40705b0 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:52.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.523+0000 7f4afa8c1640 10 monclient(hunting): authenticate will time out at 7572.786107s 2024-03-25T18:18:52.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.530+0000 7f4af8e5e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4af4149ae0 0x7f4af41e2ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:47340/0 (socket says 172.21.15.132:47340) 2024-03-25T18:18:52.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.530+0000 7f4af8e5e640 1 -- 172.21.15.132:0/2182228147 learned_addr learned my addr 172.21.15.132:0/2182228147 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:18:52.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.536+0000 7f4af0ff9640 10 client.?.objecter ms_handle_connect 0x7f4af4149ae0 2024-03-25T18:18:52.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.536+0000 7f4af0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:52.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.538+0000 7f4af0ff9640 10 client.?.objecter ms_handle_connect 0x7f4af40705b0 2024-03-25T18:18:52.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.538+0000 7f4af0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:52.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.545+0000 7f4af0ff9640 10 client.?.objecter ms_handle_connect 0x7f4af41402d0 2024-03-25T18:18:52.546 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.545+0000 7f4af0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:52.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687347448 0 0) 0x7f4af406dae0 con 0x7f4af4149ae0 2024-03-25T18:18:52.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ae0004eb0 con 0x7f4af4149ae0 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736820051 0 0) 0x7f4af406c440 con 0x7f4af40705b0 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:18:52.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.548+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af406dae0 con 0x7f4af40705b0 2024-03-25T18:18:52.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.550+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847139632 0 0) 0x7f4af406d800 con 0x7f4af41402d0 2024-03-25T18:18:52.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.550+0000 7f4af0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:52.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.550+0000 7f4af0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:52.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.550+0000 7f4af0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:52.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.550+0000 7f4af0ff9640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:18:52.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.550+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4af406c440 con 0x7f4af41402d0 2024-03-25T18:18:52.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3161593459 0 0) 0x7f4ae0004eb0 con 0x7f4af4149ae0 2024-03-25T18:18:52.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af406d800 con 0x7f4af4149ae0 2024-03-25T18:18:52.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4aec001e80 con 0x7f4af4149ae0 2024-03-25T18:18:52.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.568+0000 7f4af0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.569+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324335801 0 0) 0x7f4af406dae0 con 0x7f4af40705b0 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.569+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ae0004eb0 con 0x7f4af40705b0 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.569+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4ae4001930 con 0x7f4af40705b0 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.569+0000 7f4af0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.569+0000 7f4af0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.569+0000 7f4af0ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.571 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.572 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.572 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:52.572 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:52.572 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:52.572 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.582+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067987813 0 0) 0x7f4af406c440 con 0x7f4af41402d0 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.582+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4af406dae0 con 0x7f4af41402d0 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.582+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4ae8001830 con 0x7f4af41402d0 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.582+0000 7f4af0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.582+0000 7f4af0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.582+0000 7f4af0ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.583 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:52.584 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:52.584 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:52.584 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1759902769 0 0) 0x7f4af406d800 con 0x7f4af4149ae0 2024-03-25T18:18:52.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 >> v1:172.21.15.132:6790/0 conn(0x7f4af40705b0 legacy=0x7f4af4141910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 >> v1:172.21.15.132:6789/0 conn(0x7f4af41402d0 legacy=0x7f4af4134040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:52.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:52.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 1 monclient: found mon.b 2024-03-25T18:18:52.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4142020 con 0x7f4af4149ae0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4af0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 10 monclient: _renew_subs 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af4142270 con 0x7f4af4149ae0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 10 monclient: _renew_subs 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af406cb40 con 0x7f4af4149ae0 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.583+0000 7f4afa8c1640 1 librados: init done 2024-03-25T18:18:52.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.584+0000 7f4afa8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:52.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.584+0000 7f4afa8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:52.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.584+0000 7f4afa8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:52.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.584+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af406c2a0 con 0x7f4af4149ae0 2024-03-25T18:18:52.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4aec002080 con 0x7f4af4149ae0 2024-03-25T18:18:52.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:52.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4aec004e10 con 0x7f4af4149ae0 2024-03-25T18:18:52.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:52.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af0ff9640 10 monclient: dump: 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:18:52.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.884+0000 7f4af27fc640 4 set_mon_vals no callback set 2024-03-25T18:18:52.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.885+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f4aec01ade0 con 0x7f4af4149ae0 2024-03-25T18:18:52.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.885+0000 7f4af3fff640 10 monclient: get_auth_request con 0x7f4ae006e0f0 auth_method 0 2024-03-25T18:18:52.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.886+0000 7f4af0ff9640 10 client.4465.objecter ms_handle_connect 0x7f4ae006e0f0 2024-03-25T18:18:52.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.891+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f4aec001460 con 0x7f4af4149ae0 2024-03-25T18:18:52.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.891+0000 7f4af0ff9640 10 client.4465.objecter ms_dispatch 0x7f4af4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:52.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.891+0000 7f4af0ff9640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:52.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.891+0000 7f4af0ff9640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:52.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.891+0000 7f4af0ff9640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:18:52.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.892+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4aec050230 con 0x7f4af4149ae0 2024-03-25T18:18:52.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.892+0000 7f4af0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:52.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:52.892+0000 7f4af0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:53.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.129+0000 7f4afa8c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.129+0000 7f4afa8c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.129+0000 7f4afa8c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:18:53.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.129+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4af406c5d0 con 0x7f4af4149ae0 2024-03-25T18:18:53.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.164+0000 7f4af0ff9640 1 -- 172.21.15.132:0/2182228147 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f4aec052330 con 0x7f4af4149ae0 2024-03-25T18:18:53.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.164+0000 7f4af0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:53.165 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.164+0000 7f4af0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:53.166 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:18:53.166 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:53.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.166+0000 7f4afa8c1640 10 librados: watch_flush enter 2024-03-25T18:18:53.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 10 librados: watch_flush exit 2024-03-25T18:18:53.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:18:53.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 >> v1:172.21.15.161:6800/411950978 conn(0x7f4ae006e0f0 legacy=0x7f4ae00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 10 monclient: shutdown 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 >> v1:172.21.15.161:6789/0 conn(0x7f4af4149ae0 legacy=0x7f4af41e2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 shutdown_connections 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 >> 172.21.15.132:0/2182228147 conn(0x7f4af40b5d10 msgr2=0x7f4af413c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 shutdown_connections 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 -- 172.21.15.132:0/2182228147 wait complete. 2024-03-25T18:18:53.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:18:53.167+0000 7f4afa8c1640 1 librados: shutdown 2024-03-25T18:18:53.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:54.838 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:40.102 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.882 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:40.146 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.883 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:40.147 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.894 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:40.158 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.894 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:05:40.158 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.923 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:05:40.187 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.950 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:40.192 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.957 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:40.198 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.990 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:05:40.231 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:59.986 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:00.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.101+0000 7f23ad98f640 10 monclient: get_monmap_and_config 2024-03-25T18:19:00.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.101+0000 7f23ad98f640 10 monclient: build_initial_monmap 2024-03-25T18:19:00.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.101+0000 7f23ad98f640 10 monclient: monmap: 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:00.102642+0000 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:00.102642+0000 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:00.103 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.102+0000 7f23ad98f640 1 Processor -- start 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.102+0000 7f23ad98f640 1 -- start start 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.102+0000 7f23ad98f640 10 monclient: init 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient(hunting): picked mon.noname-a con 0x7f23a80a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient(hunting): picked mon.noname-b con 0x7f23a813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient(hunting): picked mon.noname-c con 0x7f23a81430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a8142520 con 0x7f23a80a98f0 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80b28a0 con 0x7f23a81430c0 2024-03-25T18:19:00.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80b2a30 con 0x7f23a813f710 2024-03-25T18:19:00.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.103+0000 7f23ad98f640 10 monclient(hunting): authenticate will time out at 7580.366022s 2024-03-25T18:19:00.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.108+0000 7f23a6ffd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f23a80a98f0 0x7f23a80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:35886/0 (socket says 172.21.15.132:35886) 2024-03-25T18:19:00.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.108+0000 7f23a6ffd640 1 -- 172.21.15.132:0/3354306960 learned_addr learned my addr 172.21.15.132:0/3354306960 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:00.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.140+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871203104 0 0) 0x7f23a80b2a30 con 0x7f23a813f710 2024-03-25T18:19:00.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.140+0000 7f23a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.140+0000 7f23a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.140+0000 7f23a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.140+0000 7f23a5ffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:19:00.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.140+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2390003520 con 0x7f23a813f710 2024-03-25T18:19:00.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.204+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185571556 0 0) 0x7f23a8142520 con 0x7f23a80a98f0 2024-03-25T18:19:00.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.204+0000 7f23a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.204+0000 7f23a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.204+0000 7f23a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.204+0000 7f23a5ffb640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:19:00.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.204+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a80b2a30 con 0x7f23a80a98f0 2024-03-25T18:19:00.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.212+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772513131 0 0) 0x7f23a80b28a0 con 0x7f23a81430c0 2024-03-25T18:19:00.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.212+0000 7f23a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.212+0000 7f23a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.212+0000 7f23a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.212+0000 7f23a5ffb640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:19:00.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.212+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a8142520 con 0x7f23a81430c0 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4235378402 0 0) 0x7f2390003520 con 0x7f23a813f710 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 >> v1:172.21.15.161:6789/0 conn(0x7f23a81430c0 legacy=0x7f23a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 >> v1:172.21.15.132:6789/0 conn(0x7f23a80a98f0 legacy=0x7f23a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 1 monclient: found mon.noname-b 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a80b44c0 con 0x7f23a813f710 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:00.216 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f239c001690 con 0x7f23a813f710 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23a5ffb640 10 monclient: dump: 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:00.217 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23ad98f640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:19:00.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.216+0000 7f23ad98f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:00.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f239c0017f0 con 0x7f23a813f710 2024-03-25T18:19:00.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23a5ffb640 1 -- 172.21.15.132:0/3354306960 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f239c001c00 con 0x7f23a813f710 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23a5ffb640 10 monclient: dump: 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23ad98f640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.224+0000 7f23ad98f640 4 set_mon_vals no callback set 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 10 monclient: shutdown 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 1 -- 172.21.15.132:0/3354306960 >> v1:172.21.15.132:6790/0 conn(0x7f23a813f710 legacy=0x7f23a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 1 -- 172.21.15.132:0/3354306960 shutdown_connections 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 1 -- 172.21.15.132:0/3354306960 >> 172.21.15.132:0/3354306960 conn(0x7f23a80b7ea0 msgr2=0x7f23a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 1 -- 172.21.15.132:0/3354306960 shutdown_connections 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.225+0000 7f23ad98f640 1 -- 172.21.15.132:0/3354306960 wait complete. 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.226+0000 7f23ad98f640 10 monclient: build_initial_monmap 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.226+0000 7f23ad98f640 10 monclient: monmap: 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:00.229 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.226+0000 7f23ad98f640 1 librados: starting msgr at 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.226+0000 7f23ad98f640 1 librados: starting objecter 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.226+0000 7f23ad98f640 1 Processor -- start 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 1 -- start start 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 1 librados: setting wanted keys 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 1 librados: calling monclient init 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient: init 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:00.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient(hunting): picked mon.noname-c con 0x7f23a80a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient(hunting): picked mon.noname-b con 0x7f23a813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient(hunting): picked mon.noname-a con 0x7f23a81430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80b1870 con 0x7f23a81430c0 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80ac910 con 0x7f23a813f710 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a80abb00 con 0x7f23a80a98f0 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:00.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.227+0000 7f23ad98f640 10 monclient(hunting): authenticate will time out at 7580.489435s 2024-03-25T18:19:00.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.239+0000 7f23a6ffd640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f23a80a98f0 0x7f23a8138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:51624/0 (socket says 172.21.15.132:51624) 2024-03-25T18:19:00.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.239+0000 7f23a6ffd640 1 -- 172.21.15.132:0/4166513182 learned_addr learned my addr 172.21.15.132:0/4166513182 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:00.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.240+0000 7f238ffff640 10 client.?.objecter ms_handle_connect 0x7f23a80a98f0 2024-03-25T18:19:00.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.240+0000 7f238ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.241+0000 7f238ffff640 10 client.?.objecter ms_handle_connect 0x7f23a81430c0 2024-03-25T18:19:00.241 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.241+0000 7f238ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.245+0000 7f238ffff640 10 client.?.objecter ms_handle_connect 0x7f23a813f710 2024-03-25T18:19:00.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.245+0000 7f238ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:00.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.254+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549865921 0 0) 0x7f23a80ac910 con 0x7f23a813f710 2024-03-25T18:19:00.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.254+0000 7f238ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.254+0000 7f238ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.254+0000 7f238ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.254+0000 7f238ffff640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:19:00.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.254+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394004eb0 con 0x7f23a813f710 2024-03-25T18:19:00.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.256+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472117905 0 0) 0x7f23a8142520 con 0x7f23a81430c0 2024-03-25T18:19:00.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.256+0000 7f238ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.256+0000 7f238ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.256+0000 7f238ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.256+0000 7f238ffff640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:19:00.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.256+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a80ac910 con 0x7f23a81430c0 2024-03-25T18:19:00.259 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.259+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705410665 0 0) 0x7f23a80abb00 con 0x7f23a80a98f0 2024-03-25T18:19:00.259 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.259+0000 7f238ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:00.259 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.259+0000 7f238ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:00.259 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.259+0000 7f238ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:00.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.259+0000 7f238ffff640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:19:00.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.259+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a8142520 con 0x7f23a80a98f0 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 65990674 0 0) 0x7f23a80ac910 con 0x7f23a81430c0 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a80abb00 con 0x7f23a81430c0 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f23a00019b0 con 0x7f23a81430c0 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.268 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.269 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.269 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:00.269 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:00.269 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:00.269 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.269 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.268+0000 7f238ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:00.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.270+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894933576 0 0) 0x7f2394004eb0 con 0x7f23a813f710 2024-03-25T18:19:00.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.270+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a80ac910 con 0x7f23a813f710 2024-03-25T18:19:00.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.270+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f239c001b00 con 0x7f23a813f710 2024-03-25T18:19:00.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.270+0000 7f238ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.270+0000 7f238ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.270+0000 7f238ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:00.271 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.273+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2991330890 0 0) 0x7f23a8142520 con 0x7f23a80a98f0 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.273+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2394004eb0 con 0x7f23a80a98f0 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.273+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f23980018f0 con 0x7f23a80a98f0 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.273+0000 7f238ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.273+0000 7f238ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.273+0000 7f238ffff640 10 monclient(hunting): dump: 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:00.273 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:00.274 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2639796518 0 0) 0x7f23a80ac910 con 0x7f23a813f710 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 >> v1:172.21.15.132:6790/0 conn(0x7f23a80a98f0 legacy=0x7f23a8138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 >> v1:172.21.15.132:6789/0 conn(0x7f23a81430c0 legacy=0x7f23a81e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 1 monclient: found mon.b 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a80abde0 con 0x7f23a813f710 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f238ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:19:00.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 10 monclient: _renew_subs 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a8138f40 con 0x7f23a813f710 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 10 monclient: _renew_subs 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a81363f0 con 0x7f23a813f710 2024-03-25T18:19:00.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.291+0000 7f23ad98f640 1 librados: init done 2024-03-25T18:19:00.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.292+0000 7f23ad98f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:00.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.292+0000 7f23ad98f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:00.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.292+0000 7f23ad98f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:00.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.292+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a80627b0 con 0x7f23a813f710 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f239c0038a0 con 0x7f23a813f710 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f238ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f239c003bb0 con 0x7f23a813f710 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f238ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f238ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f238ffff640 10 monclient: dump: 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:00.573 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:00.574 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:00.574 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:00.574 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:00.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.572+0000 7f23a57fa640 4 set_mon_vals no callback set 2024-03-25T18:19:00.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.573+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f239c019b80 con 0x7f23a813f710 2024-03-25T18:19:00.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.574+0000 7f23a6ffd640 10 monclient: get_auth_request con 0x7f239406e160 auth_method 0 2024-03-25T18:19:00.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.574+0000 7f238ffff640 10 client.4471.objecter ms_handle_connect 0x7f239406e160 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f239c053690 con 0x7f23a813f710 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 10 client.4471.objecter ms_dispatch 0x7f23a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f239c04f1a0 con 0x7f23a813f710 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:00.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.582+0000 7f238ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:00.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.828+0000 7f23ad98f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.828+0000 7f23ad98f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.828+0000 7f23ad98f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:00.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.828+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a81434a0 con 0x7f23a813f710 2024-03-25T18:19:00.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.851+0000 7f238ffff640 1 -- 172.21.15.132:0/4166513182 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f239c0510d0 con 0x7f23a813f710 2024-03-25T18:19:00.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.851+0000 7f238ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:00.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.851+0000 7f238ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:00.852 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:00.852 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.853+0000 7f23ad98f640 10 librados: watch_flush enter 2024-03-25T18:19:00.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 10 librados: watch_flush exit 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 10 client.4471.objecter successfully canceled tick 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 >> v1:172.21.15.161:6800/411950978 conn(0x7f239406e160 legacy=0x7f2394070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 10 monclient: shutdown 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 >> v1:172.21.15.161:6789/0 conn(0x7f23a813f710 legacy=0x7f23a80ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 shutdown_connections 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 >> 172.21.15.132:0/4166513182 conn(0x7f23a80b7ea0 msgr2=0x7f23a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 shutdown_connections 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 -- 172.21.15.132:0/4166513182 wait complete. 2024-03-25T18:19:00.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:00.854+0000 7f23ad98f640 1 librados: shutdown 2024-03-25T18:19:00.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:07.673 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:07.787 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.786+0000 7f7c375bc640 10 monclient: get_monmap_and_config 2024-03-25T18:19:07.787 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.786+0000 7f7c375bc640 10 monclient: build_initial_monmap 2024-03-25T18:19:07.787 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.786+0000 7f7c375bc640 10 monclient: monmap: 2024-03-25T18:19:07.787 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:07.787 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:07.787 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:07.787577+0000 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:07.787577+0000 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:07.788 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:07.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.787+0000 7f7c375bc640 1 Processor -- start 2024-03-25T18:19:07.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.787+0000 7f7c375bc640 1 -- start start 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.787+0000 7f7c375bc640 10 monclient: init 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient(hunting): picked mon.noname-c con 0x7f7c300a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient(hunting): picked mon.noname-b con 0x7f7c301430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c30142520 con 0x7f7c3013f710 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c300b28a0 con 0x7f7c300a98f0 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c300b2a30 con 0x7f7c301430c0 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:07.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.788+0000 7f7c375bc640 10 monclient(hunting): authenticate will time out at 7588.051240s 2024-03-25T18:19:07.802 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.802+0000 7f7c35358640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7c300a98f0 0x7f7c300a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:35616/0 (socket says 172.21.15.132:35616) 2024-03-25T18:19:07.803 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.802+0000 7f7c35358640 1 -- 172.21.15.132:0/2554239057 learned_addr learned my addr 172.21.15.132:0/2554239057 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:07.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.819+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601929993 0 0) 0x7f7c300b2a30 con 0x7f7c301430c0 2024-03-25T18:19:07.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.819+0000 7f7c27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.819+0000 7f7c27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.819+0000 7f7c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.819+0000 7f7c27fff640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:19:07.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.819+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c1c003520 con 0x7f7c301430c0 2024-03-25T18:19:07.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.824+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462496418 0 0) 0x7f7c30142520 con 0x7f7c3013f710 2024-03-25T18:19:07.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.824+0000 7f7c27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.824+0000 7f7c27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.824+0000 7f7c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.824+0000 7f7c27fff640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:19:07.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.824+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c300b2a30 con 0x7f7c3013f710 2024-03-25T18:19:07.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.826+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977331214 0 0) 0x7f7c300b28a0 con 0x7f7c300a98f0 2024-03-25T18:19:07.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.826+0000 7f7c27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.826+0000 7f7c27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.826+0000 7f7c27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.826+0000 7f7c27fff640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:19:07.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.826+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c30142520 con 0x7f7c300a98f0 2024-03-25T18:19:07.838 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3357144753 0 0) 0x7f7c30142520 con 0x7f7c300a98f0 2024-03-25T18:19:07.838 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 >> v1:172.21.15.132:6790/0 conn(0x7f7c301430c0 legacy=0x7f7c3014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 >> v1:172.21.15.132:6789/0 conn(0x7f7c3013f710 legacy=0x7f7c301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 1 monclient: found mon.noname-c 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300b44c0 con 0x7f7c300a98f0 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7c2c0015b0 con 0x7f7c300a98f0 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c375bc640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c375bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:07.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c27fff640 10 monclient: dump: 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:07.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.837+0000 7f7c375bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:07.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c2c001710 con 0x7f7c300a98f0 2024-03-25T18:19:07.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c27fff640 1 -- 172.21.15.132:0/2554239057 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7c2c001b60 con 0x7f7c300a98f0 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c375bc640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c375bc640 4 set_mon_vals no callback set 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.850+0000 7f7c27fff640 10 monclient: dump: 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:07.852 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:07.853 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 10 monclient: shutdown 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 1 -- 172.21.15.132:0/2554239057 >> v1:172.21.15.161:6789/0 conn(0x7f7c300a98f0 legacy=0x7f7c300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 1 -- 172.21.15.132:0/2554239057 shutdown_connections 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 1 -- 172.21.15.132:0/2554239057 >> 172.21.15.132:0/2554239057 conn(0x7f7c300b7ea0 msgr2=0x7f7c300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 1 -- 172.21.15.132:0/2554239057 shutdown_connections 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.851+0000 7f7c375bc640 1 -- 172.21.15.132:0/2554239057 wait complete. 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 10 monclient: build_initial_monmap 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 10 monclient: monmap: 2024-03-25T18:19:07.855 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 1 librados: starting msgr at 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 1 librados: starting objecter 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 1 Processor -- start 2024-03-25T18:19:07.856 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 1 -- start start 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 1 librados: setting wanted keys 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 1 librados: calling monclient init 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.852+0000 7f7c375bc640 10 monclient: init 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient(hunting): picked mon.noname-c con 0x7f7c300a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient(hunting): picked mon.noname-a con 0x7f7c3013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient(hunting): picked mon.noname-b con 0x7f7c301430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c301d7590 con 0x7f7c3013f710 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c300abe50 con 0x7f7c301430c0 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c301d51f0 con 0x7f7c300a98f0 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:07.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.853+0000 7f7c375bc640 10 monclient(hunting): authenticate will time out at 7588.116196s 2024-03-25T18:19:07.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.860+0000 7f7c35358640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f7c300a98f0 0x7f7c301d7d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:57222/0 (socket says 172.21.15.132:57222) 2024-03-25T18:19:07.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.860+0000 7f7c35358640 1 -- 172.21.15.132:0/774826867 learned_addr learned my addr 172.21.15.132:0/774826867 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:07.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.861+0000 7f7c25ffb640 10 client.?.objecter ms_handle_connect 0x7f7c300a98f0 2024-03-25T18:19:07.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.861+0000 7f7c25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:07.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.868+0000 7f7c25ffb640 10 client.?.objecter ms_handle_connect 0x7f7c3013f710 2024-03-25T18:19:07.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.868+0000 7f7c25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:07.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.881+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039855931 0 0) 0x7f7c301d51f0 con 0x7f7c300a98f0 2024-03-25T18:19:07.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.881+0000 7f7c25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.881+0000 7f7c25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.881+0000 7f7c25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.881+0000 7f7c25ffb640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:19:07.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.881+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c14004eb0 con 0x7f7c300a98f0 2024-03-25T18:19:07.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.882+0000 7f7c25ffb640 10 client.?.objecter ms_handle_connect 0x7f7c301430c0 2024-03-25T18:19:07.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.882+0000 7f7c25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:07.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.888+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771802566 0 0) 0x7f7c300abe50 con 0x7f7c301430c0 2024-03-25T18:19:07.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.888+0000 7f7c25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.888+0000 7f7c25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.888+0000 7f7c25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.888+0000 7f7c25ffb640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:19:07.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.888+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c301d51f0 con 0x7f7c301430c0 2024-03-25T18:19:07.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962982146 0 0) 0x7f7c301d7590 con 0x7f7c3013f710 2024-03-25T18:19:07.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:07.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:07.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:07.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:19:07.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c300abe50 con 0x7f7c3013f710 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3342238599 0 0) 0x7f7c301d51f0 con 0x7f7c301430c0 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c301d7590 con 0x7f7c301430c0 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7c28001ab0 con 0x7f7c301430c0 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.897 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.898 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:07.898 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:07.898 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:07.898 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:07.898 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:07.898 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.895+0000 7f7c25ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:07.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.901+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242747726 0 0) 0x7f7c14004eb0 con 0x7f7c300a98f0 2024-03-25T18:19:07.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.901+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c301d51f0 con 0x7f7c300a98f0 2024-03-25T18:19:07.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.901+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7c2c001ae0 con 0x7f7c300a98f0 2024-03-25T18:19:07.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.901+0000 7f7c25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:07.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.901+0000 7f7c25ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:07.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.901+0000 7f7c25ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:07.903 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:07.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.907+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 240921775 0 0) 0x7f7c301d7590 con 0x7f7c301430c0 2024-03-25T18:19:07.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.907+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 >> v1:172.21.15.132:6790/0 conn(0x7f7c300a98f0 legacy=0x7f7c301d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.907+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 >> v1:172.21.15.132:6789/0 conn(0x7f7c3013f710 legacy=0x7f7c30138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 1 monclient: found mon.b 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c301d8410 con 0x7f7c301430c0 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 10 monclient: _renew_subs 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c301d77e0 con 0x7f7c301430c0 2024-03-25T18:19:07.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 10 monclient: _renew_subs 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c30138160 con 0x7f7c301430c0 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.908+0000 7f7c375bc640 1 librados: init done 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.909+0000 7f7c375bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.909+0000 7f7c375bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.909+0000 7f7c375bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:07.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:07.909+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c300627b0 con 0x7f7c301430c0 2024-03-25T18:19:08.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c28001f80 con 0x7f7c301430c0 2024-03-25T18:19:08.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:08.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f7c28003b30 con 0x7f7c301430c0 2024-03-25T18:19:08.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c25ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c25ffb640 10 monclient: dump: 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:08.200 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:08.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.198+0000 7f7c277fe640 4 set_mon_vals no callback set 2024-03-25T18:19:08.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.199+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f7c28019b00 con 0x7f7c301430c0 2024-03-25T18:19:08.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.199+0000 7f7c35358640 10 monclient: get_auth_request con 0x7f7c1406dec0 auth_method 0 2024-03-25T18:19:08.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.200+0000 7f7c25ffb640 10 client.4477.objecter ms_handle_connect 0x7f7c1406dec0 2024-03-25T18:19:08.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f7c28053750 con 0x7f7c301430c0 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 10 client.4477.objecter ms_dispatch 0x7f7c30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c2804f260 con 0x7f7c301430c0 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.203+0000 7f7c25ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:08.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.463+0000 7f7c375bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:08.465 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.463+0000 7f7c375bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:08.465 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.463+0000 7f7c375bc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:08.465 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.463+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c301d7970 con 0x7f7c301430c0 2024-03-25T18:19:08.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.492+0000 7f7c25ffb640 1 -- 172.21.15.132:0/774826867 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f7c28051190 con 0x7f7c301430c0 2024-03-25T18:19:08.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.492+0000 7f7c25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:08.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.492+0000 7f7c25ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:08.494 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:08.494 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 10 librados: watch_flush enter 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 10 librados: watch_flush exit 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 >> v1:172.21.15.161:6800/411950978 conn(0x7f7c1406dec0 legacy=0x7f7c14070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 10 monclient: shutdown 2024-03-25T18:19:08.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:08.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 >> v1:172.21.15.161:6789/0 conn(0x7f7c301430c0 legacy=0x7f7c301e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 shutdown_connections 2024-03-25T18:19:08.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 >> 172.21.15.132:0/774826867 conn(0x7f7c300b7ea0 msgr2=0x7f7c30074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:08.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 shutdown_connections 2024-03-25T18:19:08.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 -- 172.21.15.132:0/774826867 wait complete. 2024-03-25T18:19:08.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:08.495+0000 7f7c375bc640 1 librados: shutdown 2024-03-25T18:19:08.508 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:14.943 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:00.207 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.949 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:00.213 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.976 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:00.240 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.996 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:00.237 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.998 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:00.262 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:15.003 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:00.244 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:15.010 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:00.274 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:15.048 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:00.290 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:15.049 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:00.313 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:15.314 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.428+0000 7fae5ce43640 10 monclient: get_monmap_and_config 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.428+0000 7fae5ce43640 10 monclient: build_initial_monmap 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.428+0000 7fae5ce43640 10 monclient: monmap: 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:15.429914+0000 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:15.429914+0000 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:15.430 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:15.431 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:15.431 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.429+0000 7fae5ce43640 1 Processor -- start 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.429+0000 7fae5ce43640 1 -- start start 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.429+0000 7fae5ce43640 10 monclient: init 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient(hunting): picked mon.noname-c con 0x7fae580a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient(hunting): picked mon.noname-b con 0x7fae5813f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient(hunting): picked mon.noname-a con 0x7fae581430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae58142520 con 0x7fae581430c0 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580b28a0 con 0x7fae580a98f0 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580b2a30 con 0x7fae5813f710 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.430+0000 7fae5ce43640 10 monclient(hunting): authenticate will time out at 7595.693321s 2024-03-25T18:19:15.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.442+0000 7fae5659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fae580a98f0 0x7fae580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:53908/0 (socket says 172.21.15.132:53908) 2024-03-25T18:19:15.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.442+0000 7fae5659c640 1 -- 172.21.15.132:0/2297473799 learned_addr learned my addr 172.21.15.132:0/2297473799 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:15.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.460+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998520712 0 0) 0x7fae580b28a0 con 0x7fae580a98f0 2024-03-25T18:19:15.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.460+0000 7fae5559a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.460+0000 7fae5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.460+0000 7fae5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.460+0000 7fae5559a640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:19:15.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.460+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae40003520 con 0x7fae580a98f0 2024-03-25T18:19:15.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.468+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353117346 0 0) 0x7fae580b2a30 con 0x7fae5813f710 2024-03-25T18:19:15.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.468+0000 7fae5559a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.468+0000 7fae5559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.468+0000 7fae5559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.468+0000 7fae5559a640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:19:15.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.468+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae580b28a0 con 0x7fae5813f710 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.473+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2865194285 0 0) 0x7fae40003520 con 0x7fae580a98f0 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 >> v1:172.21.15.132:6790/0 conn(0x7fae5813f710 legacy=0x7fae581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 >> v1:172.21.15.132:6789/0 conn(0x7fae581430c0 legacy=0x7fae5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 1 monclient: found mon.noname-c 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:15.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580b44c0 con 0x7fae580a98f0 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: _finish_auth 0 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5ce43640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5ce43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae48001690 con 0x7fae580a98f0 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5559a640 10 monclient: dump: 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.476 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.477 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:15.477 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:15.477 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:15.477 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:15.477 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:15.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.474+0000 7fae5ce43640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae48001870 con 0x7fae580a98f0 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5559a640 1 -- 172.21.15.132:0/2297473799 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae48001690 con 0x7fae580a98f0 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5559a640 10 monclient: dump: 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.489 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5ce43640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:15.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.488+0000 7fae5ce43640 4 set_mon_vals no callback set 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 10 monclient: shutdown 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 1 -- 172.21.15.132:0/2297473799 >> v1:172.21.15.161:6789/0 conn(0x7fae580a98f0 legacy=0x7fae580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 1 -- 172.21.15.132:0/2297473799 shutdown_connections 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 1 -- 172.21.15.132:0/2297473799 >> 172.21.15.132:0/2297473799 conn(0x7fae580b7ea0 msgr2=0x7fae580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 1 -- 172.21.15.132:0/2297473799 shutdown_connections 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.489+0000 7fae5ce43640 1 -- 172.21.15.132:0/2297473799 wait complete. 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.490+0000 7fae5ce43640 10 monclient: build_initial_monmap 2024-03-25T18:19:15.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.490+0000 7fae5ce43640 10 monclient: monmap: 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:15.492 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.490+0000 7fae5ce43640 1 librados: starting msgr at 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.490+0000 7fae5ce43640 1 librados: starting objecter 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 1 Processor -- start 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 1 -- start start 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 1 librados: setting wanted keys 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 1 librados: calling monclient init 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 10 monclient: init 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.491+0000 7fae5ce43640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 10 monclient(hunting): picked mon.noname-b con 0x7fae580a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 10 monclient(hunting): picked mon.noname-a con 0x7fae5813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:15.494 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 10 monclient(hunting): picked mon.noname-c con 0x7fae581430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:15.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae58138ef0 con 0x7fae5813f710 2024-03-25T18:19:15.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580abc30 con 0x7fae580a98f0 2024-03-25T18:19:15.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae580abea0 con 0x7fae581430c0 2024-03-25T18:19:15.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:15.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.492+0000 7fae5ce43640 10 monclient(hunting): authenticate will time out at 7595.754972s 2024-03-25T18:19:15.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.505+0000 7fae5659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fae580a98f0 0x7fae580ac2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:53954/0 (socket says 172.21.15.132:53954) 2024-03-25T18:19:15.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.505+0000 7fae5659c640 1 -- 172.21.15.132:0/3423969569 learned_addr learned my addr 172.21.15.132:0/3423969569 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:15.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.508+0000 7fae46ffd640 10 client.?.objecter ms_handle_connect 0x7fae5813f710 2024-03-25T18:19:15.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.508+0000 7fae46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.509+0000 7fae46ffd640 10 client.?.objecter ms_handle_connect 0x7fae580a98f0 2024-03-25T18:19:15.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.509+0000 7fae46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.516+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856376509 0 0) 0x7fae580abc30 con 0x7fae580a98f0 2024-03-25T18:19:15.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.516+0000 7fae46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.516+0000 7fae46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.516+0000 7fae46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.516+0000 7fae46ffd640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:19:15.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.516+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae38004eb0 con 0x7fae580a98f0 2024-03-25T18:19:15.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.523+0000 7fae46ffd640 10 client.?.objecter ms_handle_connect 0x7fae581430c0 2024-03-25T18:19:15.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.523+0000 7fae46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:15.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.525+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604069793 0 0) 0x7fae58138ef0 con 0x7fae5813f710 2024-03-25T18:19:15.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.525+0000 7fae46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.525+0000 7fae46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.525+0000 7fae46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.525+0000 7fae46ffd640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:19:15.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.525+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae580abc30 con 0x7fae5813f710 2024-03-25T18:19:15.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4101353145 0 0) 0x7fae38004eb0 con 0x7fae580a98f0 2024-03-25T18:19:15.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae58138ef0 con 0x7fae580a98f0 2024-03-25T18:19:15.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae480047e0 con 0x7fae580a98f0 2024-03-25T18:19:15.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:15.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.536+0000 7fae46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:15.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.543+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490285720 0 0) 0x7fae580abea0 con 0x7fae581430c0 2024-03-25T18:19:15.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.544+0000 7fae46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:15.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.544+0000 7fae46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:15.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.544+0000 7fae46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:15.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.544+0000 7fae46ffd640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:19:15.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.544+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae38004eb0 con 0x7fae581430c0 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.549+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2056520128 0 0) 0x7fae58138ef0 con 0x7fae580a98f0 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 >> v1:172.21.15.132:6790/0 conn(0x7fae581430c0 legacy=0x7fae58136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 >> v1:172.21.15.132:6789/0 conn(0x7fae5813f710 legacy=0x7fae58138620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 1 monclient: found mon.b 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:15.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae58139970 con 0x7fae580a98f0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 10 monclient: _renew_subs 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae58137b40 con 0x7fae580a98f0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 10 monclient: _renew_subs 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae581e3480 con 0x7fae580a98f0 2024-03-25T18:19:15.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.550+0000 7fae5ce43640 1 librados: init done 2024-03-25T18:19:15.553 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.551+0000 7fae5ce43640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.553 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.551+0000 7fae5ce43640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.553 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.551+0000 7fae5ce43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:15.553 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.551+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae580627b0 con 0x7fae580a98f0 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae48004bf0 con 0x7fae580a98f0 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fae48004f00 con 0x7fae580a98f0 2024-03-25T18:19:15.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae46ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae46ffd640 10 monclient: dump: 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:15.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.561+0000 7fae54d99640 4 set_mon_vals no callback set 2024-03-25T18:19:15.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.831+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fae4801aed0 con 0x7fae580a98f0 2024-03-25T18:19:15.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.832+0000 7fae55d9b640 10 monclient: get_auth_request con 0x7fae3806de00 auth_method 0 2024-03-25T18:19:15.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.832+0000 7fae46ffd640 10 client.4483.objecter ms_handle_connect 0x7fae3806de00 2024-03-25T18:19:15.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.835+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fae480549a0 con 0x7fae580a98f0 2024-03-25T18:19:15.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.835+0000 7fae46ffd640 10 client.4483.objecter ms_dispatch 0x7fae58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:15.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.835+0000 7fae46ffd640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:15.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.835+0000 7fae46ffd640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:15.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.835+0000 7fae46ffd640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T18:19:15.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.838+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae480504b0 con 0x7fae580a98f0 2024-03-25T18:19:15.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.838+0000 7fae46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:15.839 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:15.838+0000 7fae46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:16.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.105+0000 7fae5ce43640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.105+0000 7fae5ce43640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.105+0000 7fae5ce43640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:16.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.105+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae5813e1c0 con 0x7fae580a98f0 2024-03-25T18:19:16.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.247+0000 7fae46ffd640 1 -- 172.21.15.132:0/3423969569 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fae480523e0 con 0x7fae580a98f0 2024-03-25T18:19:16.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.247+0000 7fae46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:16.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.247+0000 7fae46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:16.248 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:16.248 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.249+0000 7fae5ce43640 10 librados: watch_flush enter 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.249+0000 7fae5ce43640 10 librados: watch_flush exit 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.249+0000 7fae5ce43640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.249+0000 7fae5ce43640 10 client.4483.objecter successfully canceled tick 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 >> v1:172.21.15.161:6800/411950978 conn(0x7fae3806de00 legacy=0x7fae380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 10 monclient: shutdown 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 >> v1:172.21.15.161:6789/0 conn(0x7fae580a98f0 legacy=0x7fae580ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 shutdown_connections 2024-03-25T18:19:16.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 >> 172.21.15.132:0/3423969569 conn(0x7fae580b7ea0 msgr2=0x7fae58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 shutdown_connections 2024-03-25T18:19:16.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 -- 172.21.15.132:0/3423969569 wait complete. 2024-03-25T18:19:16.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:16.250+0000 7fae5ce43640 1 librados: shutdown 2024-03-25T18:19:16.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:23.068 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.167+0000 7f76209e8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.167+0000 7f76209e8640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.168+0000 7f76209e8640 10 monclient: monmap: 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:23.169048+0000 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:23.169048+0000 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:23.169 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:23.170 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:23.170 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.168+0000 7f76209e8640 1 Processor -- start 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.168+0000 7f76209e8640 1 -- start start 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.168+0000 7f76209e8640 10 monclient: init 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient(hunting): picked mon.noname-c con 0x7f761c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient(hunting): picked mon.noname-b con 0x7f761c13f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient(hunting): picked mon.noname-a con 0x7f761c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c142520 con 0x7f761c1430c0 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0b28a0 con 0x7f761c0a98f0 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0b2a30 con 0x7f761c13f710 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.169+0000 7f76209e8640 10 monclient(hunting): authenticate will time out at 7603.432349s 2024-03-25T18:19:23.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.182+0000 7f761a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f761c0a98f0 0x7f761c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:52194/0 (socket says 172.21.15.132:52194) 2024-03-25T18:19:23.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.182+0000 7f761a59c640 1 -- 172.21.15.132:0/3008467123 learned_addr learned my addr 172.21.15.132:0/3008467123 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:23.196 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.195+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597421641 0 0) 0x7f761c0b28a0 con 0x7f761c0a98f0 2024-03-25T18:19:23.196 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.195+0000 7f761959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.196 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.195+0000 7f761959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.196 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.195+0000 7f761959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.196 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.195+0000 7f761959a640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:19:23.196 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.195+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7604003520 con 0x7f761c0a98f0 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156667720 0 0) 0x7f761c0b2a30 con 0x7f761c13f710 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c0b28a0 con 0x7f761c13f710 2024-03-25T18:19:23.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523459351 0 0) 0x7f761c142520 con 0x7f761c1430c0 2024-03-25T18:19:23.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:19:23.204 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.202+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c0b2a30 con 0x7f761c1430c0 2024-03-25T18:19:23.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.208+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808881238 0 0) 0x7f7604003520 con 0x7f761c0a98f0 2024-03-25T18:19:23.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.208+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 >> v1:172.21.15.132:6790/0 conn(0x7f761c13f710 legacy=0x7f761c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.208+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 >> v1:172.21.15.132:6789/0 conn(0x7f761c1430c0 legacy=0x7f761c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:23.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 1 monclient: found mon.noname-c 2024-03-25T18:19:23.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c0b44c0 con 0x7f761c0a98f0 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: _finish_auth 0 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f760c001690 con 0x7f761c0a98f0 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f76209e8640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f76209e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f761959a640 10 monclient: dump: 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:23.211 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:23.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.209+0000 7f76209e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:23.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f760c0017f0 con 0x7f761c0a98f0 2024-03-25T18:19:23.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f761959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:23.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f761959a640 1 -- 172.21.15.132:0/3008467123 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f760c001c00 con 0x7f761c0a98f0 2024-03-25T18:19:23.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f76209e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:23.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f76209e8640 4 set_mon_vals no callback set 2024-03-25T18:19:23.239 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f761959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f761959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.238+0000 7f761959a640 10 monclient: dump: 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:23.240 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 10 monclient: shutdown 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 1 -- 172.21.15.132:0/3008467123 >> v1:172.21.15.161:6789/0 conn(0x7f761c0a98f0 legacy=0x7f761c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 1 -- 172.21.15.132:0/3008467123 shutdown_connections 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 1 -- 172.21.15.132:0/3008467123 >> 172.21.15.132:0/3008467123 conn(0x7f761c0b7ea0 msgr2=0x7f761c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 1 -- 172.21.15.132:0/3008467123 shutdown_connections 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.239+0000 7f76209e8640 1 -- 172.21.15.132:0/3008467123 wait complete. 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.240+0000 7f76209e8640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.240+0000 7f76209e8640 10 monclient: monmap: 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:23.242 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 1 librados: starting msgr at 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 1 librados: starting objecter 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 1 Processor -- start 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 1 -- start start 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 1 librados: setting wanted keys 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 1 librados: calling monclient init 2024-03-25T18:19:23.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.241+0000 7f76209e8640 10 monclient: init 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient(hunting): picked mon.noname-b con 0x7f761c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient(hunting): picked mon.noname-a con 0x7f761c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient(hunting): picked mon.noname-c con 0x7f761c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c139870 con 0x7f761c13f710 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c1d51f0 con 0x7f761c0a98f0 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c138830 con 0x7f761c1430c0 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.242+0000 7f76209e8640 10 monclient(hunting): authenticate will time out at 7603.505072s 2024-03-25T18:19:23.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.249+0000 7f761a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f761c0a98f0 0x7f761c138120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:52256/0 (socket says 172.21.15.132:52256) 2024-03-25T18:19:23.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.249+0000 7f761a59c640 1 -- 172.21.15.132:0/2692832236 learned_addr learned my addr 172.21.15.132:0/2692832236 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:23.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.253+0000 7f760affd640 10 client.?.objecter ms_handle_connect 0x7f761c0a98f0 2024-03-25T18:19:23.254 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.253+0000 7f760affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.259+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449293295 0 0) 0x7f761c1d51f0 con 0x7f761c0a98f0 2024-03-25T18:19:23.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.259+0000 7f760affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.259+0000 7f760affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.259+0000 7f760affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.259+0000 7f760affd640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:19:23.260 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.259+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc004eb0 con 0x7f761c0a98f0 2024-03-25T18:19:23.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.264+0000 7f760affd640 10 client.?.objecter ms_handle_connect 0x7f761c1430c0 2024-03-25T18:19:23.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.264+0000 7f760affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.271+0000 7f760affd640 10 client.?.objecter ms_handle_connect 0x7f761c13f710 2024-03-25T18:19:23.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.271+0000 7f760affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 728011406 0 0) 0x7f75fc004eb0 con 0x7f761c0a98f0 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f761c1d51f0 con 0x7f761c0a98f0 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f760c001bc0 con 0x7f761c0a98f0 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 10 monclient(hunting): dump: 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:23.273 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:23.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.272+0000 7f760affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798693475 0 0) 0x7f761c138830 con 0x7f761c1430c0 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75fc004eb0 con 0x7f761c1430c0 2024-03-25T18:19:23.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483602094 0 0) 0x7f761c139870 con 0x7f761c13f710 2024-03-25T18:19:23.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:23.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:23.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:23.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:19:23.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.283+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c138830 con 0x7f761c13f710 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1047647526 0 0) 0x7f761c1d51f0 con 0x7f761c0a98f0 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 >> v1:172.21.15.132:6790/0 conn(0x7f761c1430c0 legacy=0x7f761c1e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 >> v1:172.21.15.132:6789/0 conn(0x7f761c13f710 legacy=0x7f761c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 1 monclient: found mon.b 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c1d77a0 con 0x7f761c0a98f0 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:23.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f760affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 5 monclient: authenticate success, global_id 4489 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 10 monclient: _renew_subs 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c1d83d0 con 0x7f761c0a98f0 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 10 client.4489.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 10 monclient: _renew_subs 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c138c80 con 0x7f761c0a98f0 2024-03-25T18:19:23.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.288+0000 7f76209e8640 1 librados: init done 2024-03-25T18:19:23.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.289+0000 7f76209e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:23.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.289+0000 7f76209e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:23.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.289+0000 7f76209e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:23.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.289+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c0627b0 con 0x7f761c0a98f0 2024-03-25T18:19:23.566 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f760c0011f0 con 0x7f761c0a98f0 2024-03-25T18:19:23.566 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f760affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f760c004c30 con 0x7f761c0a98f0 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f760affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f760affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f760affd640 10 monclient: dump: 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:23.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.565+0000 7f7618d99640 4 set_mon_vals no callback set 2024-03-25T18:19:23.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.566+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f760c004e10 con 0x7f761c0a98f0 2024-03-25T18:19:23.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.567+0000 7f7619d9b640 10 monclient: get_auth_request con 0x7f75fc06de10 auth_method 0 2024-03-25T18:19:23.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.567+0000 7f760affd640 10 client.4489.objecter ms_handle_connect 0x7f75fc06de10 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f760c054a40 con 0x7f761c0a98f0 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 10 client.4489.objecter ms_dispatch 0x7f761c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 3 client.4489.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 3 client.4489.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 20 client.4489.objecter dump_active .. 0 homeless 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f760c050380 con 0x7f761c0a98f0 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:23.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.571+0000 7f760affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:23.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.819+0000 7f76209e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.819+0000 7f76209e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.819+0000 7f76209e8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:23.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.819+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f761c1d7930 con 0x7f761c0a98f0 2024-03-25T18:19:23.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.853+0000 7f760affd640 1 -- 172.21.15.132:0/2692832236 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f760c052480 con 0x7f761c0a98f0 2024-03-25T18:19:23.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.853+0000 7f760affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:23.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.853+0000 7f760affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:23.854 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:23.855 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:23.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.855+0000 7f76209e8640 10 librados: watch_flush enter 2024-03-25T18:19:23.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.855+0000 7f76209e8640 10 librados: watch_flush exit 2024-03-25T18:19:23.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.855+0000 7f76209e8640 20 client.4489.objecter shutdown clearing up homeless session... 2024-03-25T18:19:23.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.855+0000 7f76209e8640 10 client.4489.objecter successfully canceled tick 2024-03-25T18:19:23.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 >> v1:172.21.15.161:6800/411950978 conn(0x7f75fc06de10 legacy=0x7f75fc0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.857 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 10 monclient: shutdown 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 >> v1:172.21.15.161:6789/0 conn(0x7f761c0a98f0 legacy=0x7f761c138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 shutdown_connections 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 >> 172.21.15.132:0/2692832236 conn(0x7f761c0b7ea0 msgr2=0x7f761c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 shutdown_connections 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 -- 172.21.15.132:0/2692832236 wait complete. 2024-03-25T18:19:23.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:23.856+0000 7f76209e8640 1 librados: shutdown 2024-03-25T18:19:23.868 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:30.674 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:30.788 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.787+0000 7eff3c582640 10 monclient: get_monmap_and_config 2024-03-25T18:19:30.788 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.787+0000 7eff3c582640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.788 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.787+0000 7eff3c582640 10 monclient: monmap: 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:30.788721+0000 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:30.788721+0000 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:30.789 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.788+0000 7eff3c582640 1 Processor -- start 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.788+0000 7eff3c582640 1 -- start start 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.788+0000 7eff3c582640 10 monclient: init 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient(hunting): picked mon.noname-c con 0x7eff340a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:30.790 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient(hunting): picked mon.noname-a con 0x7eff3413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:30.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient(hunting): picked mon.noname-b con 0x7eff341430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:30.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff34142520 con 0x7eff3413f710 2024-03-25T18:19:30.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff340b28a0 con 0x7eff340a98f0 2024-03-25T18:19:30.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff340b2a30 con 0x7eff341430c0 2024-03-25T18:19:30.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.789+0000 7eff3c582640 10 monclient(hunting): authenticate will time out at 7611.052135s 2024-03-25T18:19:30.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.793+0000 7eff3a31e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7eff340a98f0 0x7eff340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:56914/0 (socket says 172.21.15.132:56914) 2024-03-25T18:19:30.794 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.793+0000 7eff3a31e640 1 -- 172.21.15.132:0/2861777953 learned_addr learned my addr 172.21.15.132:0/2861777953 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:30.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.808+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51551965 0 0) 0x7eff340b28a0 con 0x7eff340a98f0 2024-03-25T18:19:30.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.808+0000 7eff3931c640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.808+0000 7eff3931c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.808+0000 7eff3931c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.808+0000 7eff3931c640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:19:30.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.808+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff20003520 con 0x7eff340a98f0 2024-03-25T18:19:30.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.810+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 500987265 0 0) 0x7eff34142520 con 0x7eff3413f710 2024-03-25T18:19:30.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.810+0000 7eff3931c640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.810+0000 7eff3931c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.810+0000 7eff3931c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.810+0000 7eff3931c640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:19:30.811 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.810+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff340b28a0 con 0x7eff3413f710 2024-03-25T18:19:30.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.817+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904166626 0 0) 0x7eff340b2a30 con 0x7eff341430c0 2024-03-25T18:19:30.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.817+0000 7eff3931c640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.817+0000 7eff3931c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.817+0000 7eff3931c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.817+0000 7eff3931c640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:19:30.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.817+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34142520 con 0x7eff341430c0 2024-03-25T18:19:30.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856383537 0 0) 0x7eff340b28a0 con 0x7eff3413f710 2024-03-25T18:19:30.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 >> v1:172.21.15.132:6790/0 conn(0x7eff341430c0 legacy=0x7eff3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 >> v1:172.21.15.161:6789/0 conn(0x7eff340a98f0 legacy=0x7eff340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: _finish_hunting 0 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 1 monclient: found mon.noname-a 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340b44c0 con 0x7eff3413f710 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: _finish_auth 0 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: _check_auth_tickets 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7eff24001670 con 0x7eff3413f710 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3c582640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3c582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3931c640 10 monclient: dump: 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:30.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.820+0000 7eff3c582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:30.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff240017d0 con 0x7eff3413f710 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3931c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3931c640 1 -- 172.21.15.132:0/2861777953 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7eff24001be0 con 0x7eff3413f710 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3c582640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3c582640 4 set_mon_vals no callback set 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3931c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3931c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.826+0000 7eff3931c640 10 monclient: dump: 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.828 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:30.829 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:30.829 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:30.829 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:30.829 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.827+0000 7eff3c582640 10 monclient: shutdown 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.827+0000 7eff3c582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.827+0000 7eff3c582640 1 -- 172.21.15.132:0/2861777953 >> v1:172.21.15.132:6789/0 conn(0x7eff3413f710 legacy=0x7eff341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.827+0000 7eff3c582640 1 -- 172.21.15.132:0/2861777953 shutdown_connections 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.827+0000 7eff3c582640 1 -- 172.21.15.132:0/2861777953 >> 172.21.15.132:0/2861777953 conn(0x7eff340b7ea0 msgr2=0x7eff340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.827+0000 7eff3c582640 1 -- 172.21.15.132:0/2861777953 shutdown_connections 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.828+0000 7eff3c582640 1 -- 172.21.15.132:0/2861777953 wait complete. 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.828+0000 7eff3c582640 10 monclient: build_initial_monmap 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.828+0000 7eff3c582640 10 monclient: monmap: 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:30.831 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 librados: starting msgr at 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 librados: starting objecter 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 Processor -- start 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 -- start start 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 librados: setting wanted keys 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 librados: calling monclient init 2024-03-25T18:19:30.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient: init 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient(hunting): picked mon.noname-c con 0x7eff340a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient(hunting): picked mon.noname-a con 0x7eff3413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient(hunting): picked mon.noname-b con 0x7eff341430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff34136750 con 0x7eff3413f710 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff341dac30 con 0x7eff341430c0 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff341399c0 con 0x7eff340a98f0 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:30.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.829+0000 7eff3c582640 10 monclient(hunting): authenticate will time out at 7611.092627s 2024-03-25T18:19:30.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.843+0000 7eff39b1d640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7eff3413f710 0x7eff341db150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:47768/0 (socket says 172.21.15.132:47768) 2024-03-25T18:19:30.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.843+0000 7eff39b1d640 1 -- 172.21.15.132:0/4065461049 learned_addr learned my addr 172.21.15.132:0/4065461049 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:30.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.843+0000 7eff2affd640 10 client.?.objecter ms_handle_connect 0x7eff3413f710 2024-03-25T18:19:30.844 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.843+0000 7eff2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.851+0000 7eff2affd640 10 client.?.objecter ms_handle_connect 0x7eff341430c0 2024-03-25T18:19:30.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.851+0000 7eff2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.858 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.858+0000 7eff2affd640 10 client.?.objecter ms_handle_connect 0x7eff340a98f0 2024-03-25T18:19:30.859 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.858+0000 7eff2affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:30.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.865+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507844924 0 0) 0x7eff341dac30 con 0x7eff341430c0 2024-03-25T18:19:30.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.866+0000 7eff2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.866+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.866+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.866+0000 7eff2affd640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:19:30.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.866+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff18004eb0 con 0x7eff341430c0 2024-03-25T18:19:30.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.868+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372544354 0 0) 0x7eff34136750 con 0x7eff3413f710 2024-03-25T18:19:30.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.868+0000 7eff2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.868+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.868+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.868+0000 7eff2affd640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:19:30.869 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.868+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff341dac30 con 0x7eff3413f710 2024-03-25T18:19:30.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.874+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184495649 0 0) 0x7eff341399c0 con 0x7eff340a98f0 2024-03-25T18:19:30.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.874+0000 7eff2affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:30.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.874+0000 7eff2affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:30.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.874+0000 7eff2affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:30.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.874+0000 7eff2affd640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:19:30.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.874+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34136750 con 0x7eff340a98f0 2024-03-25T18:19:30.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.878+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3577686198 0 0) 0x7eff18004eb0 con 0x7eff341430c0 2024-03-25T18:19:30.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.878+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff341399c0 con 0x7eff341430c0 2024-03-25T18:19:30.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.881+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7eff2c001c10 con 0x7eff341430c0 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.881+0000 7eff2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.881+0000 7eff2affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.881+0000 7eff2affd640 10 monclient(hunting): dump: 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:30.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.881+0000 7eff2affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:30.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.887+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3925676230 0 0) 0x7eff341dac30 con 0x7eff3413f710 2024-03-25T18:19:30.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.887+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff18004eb0 con 0x7eff3413f710 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.887+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7eff24001670 con 0x7eff3413f710 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.887+0000 7eff2affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.887+0000 7eff2affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.887+0000 7eff2affd640 10 monclient(hunting): dump: 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:30.889 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:30.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3030213899 0 0) 0x7eff341399c0 con 0x7eff341430c0 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 >> v1:172.21.15.132:6790/0 conn(0x7eff340a98f0 legacy=0x7eff341386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 >> v1:172.21.15.132:6789/0 conn(0x7eff3413f710 legacy=0x7eff341db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 1 monclient: found mon.b 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff341db860 con 0x7eff341430c0 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 10 monclient: _renew_subs 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:30.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff341d41e0 con 0x7eff341430c0 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 10 monclient: _renew_subs 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff341d5150 con 0x7eff341430c0 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.902+0000 7eff3c582640 1 librados: init done 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.903+0000 7eff3c582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.903+0000 7eff3c582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:30.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.903+0000 7eff3c582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:30.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:30.903+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff341daa90 con 0x7eff341430c0 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff2c0020e0 con 0x7eff341430c0 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff2affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7eff2c003990 con 0x7eff341430c0 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff2affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff2affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff2affd640 10 monclient: dump: 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:31.176 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:31.177 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:31.177 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:31.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.174+0000 7eff38b1b640 4 set_mon_vals no callback set 2024-03-25T18:19:31.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.176+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7eff2c0199e0 con 0x7eff341430c0 2024-03-25T18:19:31.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.176+0000 7eff3a31e640 10 monclient: get_auth_request con 0x7eff1806df60 auth_method 0 2024-03-25T18:19:31.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.177+0000 7eff2affd640 10 client.4495.objecter ms_handle_connect 0x7eff1806df60 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7eff2c053780 con 0x7eff341430c0 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 10 client.4495.objecter ms_dispatch 0x7eff34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff2c04f0c0 con 0x7eff341430c0 2024-03-25T18:19:31.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.180+0000 7eff2affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:31.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.420+0000 7eff3c582640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:31.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.420+0000 7eff3c582640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:31.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.420+0000 7eff3c582640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:31.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.420+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff341dadc0 con 0x7eff341430c0 2024-03-25T18:19:31.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.532+0000 7eff2affd640 1 -- 172.21.15.132:0/4065461049 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7eff2c0511c0 con 0x7eff341430c0 2024-03-25T18:19:31.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.533+0000 7eff2affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:31.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.533+0000 7eff2affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:31.534 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:31.534 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:31.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 10 librados: watch_flush enter 2024-03-25T18:19:31.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 10 librados: watch_flush exit 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 10 client.4495.objecter successfully canceled tick 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 >> v1:172.21.15.161:6800/411950978 conn(0x7eff1806df60 legacy=0x7eff18070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 10 monclient: shutdown 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 >> v1:172.21.15.161:6789/0 conn(0x7eff341430c0 legacy=0x7eff341d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 shutdown_connections 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 >> 172.21.15.132:0/4065461049 conn(0x7eff340b7ea0 msgr2=0x7eff34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 shutdown_connections 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 -- 172.21.15.132:0/4065461049 wait complete. 2024-03-25T18:19:31.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:31.535+0000 7eff3c582640 1 librados: shutdown 2024-03-25T18:19:31.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:35.019 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:20.283 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.026 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:20.290 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.076 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:20.317 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.077 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:20.341 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.088 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:20.330 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.100 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:20.364 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.111 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:20.353 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.118 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:20.382 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:35.141 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:20.405 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:38.354 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:38.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.471+0000 7fe637ca5640 10 monclient: get_monmap_and_config 2024-03-25T18:19:38.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.471+0000 7fe637ca5640 10 monclient: build_initial_monmap 2024-03-25T18:19:38.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.471+0000 7fe637ca5640 10 monclient: monmap: 2024-03-25T18:19:38.471 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:38.471 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:38.471 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:38.471627+0000 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:38.471627+0000 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:38.472 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:38.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.472+0000 7fe637ca5640 1 Processor -- start 2024-03-25T18:19:38.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.472+0000 7fe637ca5640 1 -- start start 2024-03-25T18:19:38.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.472+0000 7fe637ca5640 10 monclient: init 2024-03-25T18:19:38.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:38.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:38.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fe6300a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fe63013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fe6301430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe630142520 con 0x7fe63013f710 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300b28a0 con 0x7fe6300a98f0 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300b2a30 con 0x7fe6301430c0 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.473+0000 7fe637ca5640 10 monclient(hunting): authenticate will time out at 7618.735173s 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.474+0000 7fe635240640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe63013f710 0x7fe6301429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54726/0 (socket says 172.21.15.132:54726) 2024-03-25T18:19:38.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.474+0000 7fe635240640 1 -- 172.21.15.132:0/195128608 learned_addr learned my addr 172.21.15.132:0/195128608 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:38.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.483+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3016211203 0 0) 0x7fe630142520 con 0x7fe63013f710 2024-03-25T18:19:38.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.483+0000 7fe634a3f640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.483+0000 7fe634a3f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.483+0000 7fe634a3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.483+0000 7fe634a3f640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:19:38.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.483+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe61c003520 con 0x7fe63013f710 2024-03-25T18:19:38.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.488+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665430254 0 0) 0x7fe6300b2a30 con 0x7fe6301430c0 2024-03-25T18:19:38.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.488+0000 7fe634a3f640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.488+0000 7fe634a3f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.488+0000 7fe634a3f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.488+0000 7fe634a3f640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:19:38.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.488+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe630142520 con 0x7fe6301430c0 2024-03-25T18:19:38.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285660466 0 0) 0x7fe61c003520 con 0x7fe63013f710 2024-03-25T18:19:38.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 >> v1:172.21.15.132:6790/0 conn(0x7fe6301430c0 legacy=0x7fe63014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 >> v1:172.21.15.161:6789/0 conn(0x7fe6300a98f0 legacy=0x7fe6300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: _finish_hunting 0 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 1 monclient: found mon.noname-a 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300b44c0 con 0x7fe63013f710 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: _finish_auth 0 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: _check_auth_tickets 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe637ca5640 5 monclient: authenticate success, global_id 4500 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe637ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe620001650 con 0x7fe63013f710 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe634a3f640 10 monclient: dump: 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:38.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.491+0000 7fe637ca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6200018a0 con 0x7fe63013f710 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe634a3f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe634a3f640 1 -- 172.21.15.132:0/195128608 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe620001c30 con 0x7fe63013f710 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe637ca5640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe637ca5640 4 set_mon_vals no callback set 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe634a3f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe634a3f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe634a3f640 10 monclient: dump: 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.499 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.500 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.500 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:38.500 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:38.500 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:38.500 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:38.500 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:38.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe637ca5640 10 monclient: shutdown 2024-03-25T18:19:38.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe637ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:38.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.499+0000 7fe637ca5640 1 -- 172.21.15.132:0/195128608 >> v1:172.21.15.132:6789/0 conn(0x7fe63013f710 legacy=0x7fe6301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.500+0000 7fe637ca5640 1 -- 172.21.15.132:0/195128608 shutdown_connections 2024-03-25T18:19:38.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.500+0000 7fe637ca5640 1 -- 172.21.15.132:0/195128608 >> 172.21.15.132:0/195128608 conn(0x7fe6300b7ea0 msgr2=0x7fe6300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:38.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.500+0000 7fe637ca5640 1 -- 172.21.15.132:0/195128608 shutdown_connections 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.500+0000 7fe637ca5640 1 -- 172.21.15.132:0/195128608 wait complete. 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 10 monclient: build_initial_monmap 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 10 monclient: monmap: 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 1 librados: starting msgr at 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 1 librados: starting objecter 2024-03-25T18:19:38.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 1 Processor -- start 2024-03-25T18:19:38.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 1 -- start start 2024-03-25T18:19:38.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 1 librados: setting wanted keys 2024-03-25T18:19:38.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 1 librados: calling monclient init 2024-03-25T18:19:38.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.501+0000 7fe637ca5640 10 monclient: init 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient(hunting): picked mon.noname-a con 0x7fe6300a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient(hunting): picked mon.noname-c con 0x7fe63013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient(hunting): picked mon.noname-b con 0x7fe6301430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe630139410 con 0x7fe6300a98f0 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe630139170 con 0x7fe6301430c0 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6300ac0c0 con 0x7fe63013f710 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:38.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.502+0000 7fe637ca5640 10 monclient(hunting): authenticate will time out at 7618.764140s 2024-03-25T18:19:38.511 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.512+0000 7fe635240640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fe63013f710 0x7fe6300ac4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:48684/0 (socket says 172.21.15.132:48684) 2024-03-25T18:19:38.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.512+0000 7fe635240640 1 -- 172.21.15.132:0/4116426448 learned_addr learned my addr 172.21.15.132:0/4116426448 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:38.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.512+0000 7fe6267fc640 10 client.?.objecter ms_handle_connect 0x7fe63013f710 2024-03-25T18:19:38.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.512+0000 7fe6267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:38.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.514+0000 7fe6267fc640 10 client.?.objecter ms_handle_connect 0x7fe6301430c0 2024-03-25T18:19:38.514 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.514+0000 7fe6267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:38.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.516+0000 7fe6267fc640 10 client.?.objecter ms_handle_connect 0x7fe6300a98f0 2024-03-25T18:19:38.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.516+0000 7fe6267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:38.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.525+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092034187 0 0) 0x7fe630139170 con 0x7fe6301430c0 2024-03-25T18:19:38.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.525+0000 7fe6267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.525+0000 7fe6267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.525+0000 7fe6267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.525+0000 7fe6267fc640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:19:38.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.525+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe614004eb0 con 0x7fe6301430c0 2024-03-25T18:19:38.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.528+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1987332301 0 0) 0x7fe630139410 con 0x7fe6300a98f0 2024-03-25T18:19:38.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.528+0000 7fe6267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.528+0000 7fe6267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.528+0000 7fe6267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.528+0000 7fe6267fc640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:19:38.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.528+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe630139170 con 0x7fe6300a98f0 2024-03-25T18:19:38.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.531+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340752 0 0) 0x7fe6300ac0c0 con 0x7fe63013f710 2024-03-25T18:19:38.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.531+0000 7fe6267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:38.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.531+0000 7fe6267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:38.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.531+0000 7fe6267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:38.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.531+0000 7fe6267fc640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:19:38.532 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.531+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe630139410 con 0x7fe63013f710 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.548+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2531067713 0 0) 0x7fe614004eb0 con 0x7fe6301430c0 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.548+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6300ac0c0 con 0x7fe6301430c0 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.548+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe628001bd0 con 0x7fe6301430c0 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.548+0000 7fe6267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.549+0000 7fe6267fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.549+0000 7fe6267fc640 10 monclient(hunting): dump: 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.549 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:38.550 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:38.550 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:38.550 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:38.550 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:38.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.549+0000 7fe6267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:38.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4164030190 0 0) 0x7fe6300ac0c0 con 0x7fe6301430c0 2024-03-25T18:19:38.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 >> v1:172.21.15.132:6790/0 conn(0x7fe63013f710 legacy=0x7fe6300ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 >> v1:172.21.15.132:6789/0 conn(0x7fe6300a98f0 legacy=0x7fe630138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 1 monclient: found mon.b 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6300acbf0 con 0x7fe6301430c0 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe6267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 10 monclient: _renew_subs 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:38.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6300ace40 con 0x7fe6301430c0 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 10 client.4498.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 10 monclient: _renew_subs 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe630136a00 con 0x7fe6301430c0 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.561+0000 7fe637ca5640 1 librados: init done 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.562+0000 7fe637ca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:38.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.562+0000 7fe637ca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:38.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.562+0000 7fe637ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:38.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.562+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6300627b0 con 0x7fe6301430c0 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe628002080 con 0x7fe6301430c0 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe6267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe628003b30 con 0x7fe6301430c0 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe6267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe6267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe6267fc640 10 monclient: dump: 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:38.836 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.835+0000 7fe627fff640 4 set_mon_vals no callback set 2024-03-25T18:19:38.837 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.836+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fe628019bc0 con 0x7fe6301430c0 2024-03-25T18:19:38.838 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.837+0000 7fe635a41640 10 monclient: get_auth_request con 0x7fe61406ddf0 auth_method 0 2024-03-25T18:19:38.838 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.837+0000 7fe6267fc640 10 client.4498.objecter ms_handle_connect 0x7fe61406ddf0 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.842+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fe628053830 con 0x7fe6301430c0 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.842+0000 7fe6267fc640 10 client.4498.objecter ms_dispatch 0x7fe630000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.842+0000 7fe6267fc640 3 client.4498.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.842+0000 7fe6267fc640 3 client.4498.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.843+0000 7fe6267fc640 20 client.4498.objecter dump_active .. 0 homeless 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.843+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe62804f170 con 0x7fe6301430c0 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.843+0000 7fe6267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:38.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:38.843+0000 7fe6267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:39.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.099+0000 7fe637ca5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.099+0000 7fe637ca5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.099+0000 7fe637ca5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:19:39.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.099+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6301363f0 con 0x7fe6301430c0 2024-03-25T18:19:39.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.146+0000 7fe6267fc640 1 -- 172.21.15.132:0/4116426448 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fe628051270 con 0x7fe6301430c0 2024-03-25T18:19:39.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.146+0000 7fe6267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.146+0000 7fe6267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:39.147 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:39.147 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:39.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 10 librados: watch_flush enter 2024-03-25T18:19:39.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 10 librados: watch_flush exit 2024-03-25T18:19:39.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 20 client.4498.objecter shutdown clearing up homeless session... 2024-03-25T18:19:39.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 10 client.4498.objecter successfully canceled tick 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 >> v1:172.21.15.161:6800/411950978 conn(0x7fe61406ddf0 legacy=0x7fe6140702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 10 monclient: shutdown 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.148+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 >> v1:172.21.15.161:6789/0 conn(0x7fe6301430c0 legacy=0x7fe6301e2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.149+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 shutdown_connections 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.149+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 >> 172.21.15.132:0/4116426448 conn(0x7fe6300b7ea0 msgr2=0x7fe630074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.149+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 shutdown_connections 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.149+0000 7fe637ca5640 1 -- 172.21.15.132:0/4116426448 wait complete. 2024-03-25T18:19:39.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:39.149+0000 7fe637ca5640 1 librados: shutdown 2024-03-25T18:19:39.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:45.968 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.083+0000 7f6bb7a05640 10 monclient: get_monmap_and_config 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.083+0000 7f6bb7a05640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.083+0000 7f6bb7a05640 10 monclient: monmap: 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:46.084169+0000 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:46.084169+0000 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:46.084 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:46.085 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:46.085 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:46.085 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:46.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.083+0000 7f6bb7a05640 1 Processor -- start 2024-03-25T18:19:46.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 1 -- start start 2024-03-25T18:19:46.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient: init 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb00a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb01430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb0142520 con 0x7f6bb00a98f0 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00b28a0 con 0x7f6bb013f710 2024-03-25T18:19:46.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00b2a30 con 0x7f6bb01430c0 2024-03-25T18:19:46.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.084+0000 7f6bb7a05640 10 monclient(hunting): authenticate will time out at 7626.347673s 2024-03-25T18:19:46.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.094+0000 7f6bb4fa0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6bb013f710 0x7f6bb01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:55048/0 (socket says 172.21.15.132:55048) 2024-03-25T18:19:46.095 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.094+0000 7f6bb4fa0640 1 -- 172.21.15.132:0/2386587925 learned_addr learned my addr 172.21.15.132:0/2386587925 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:46.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.098+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639608194 0 0) 0x7f6bb00b28a0 con 0x7f6bb013f710 2024-03-25T18:19:46.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.098+0000 7f6ba7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.098+0000 7f6ba7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.098+0000 7f6ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.098+0000 7f6ba7fff640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:19:46.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.098+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b9c003520 con 0x7f6bb013f710 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32577371 0 0) 0x7f6b9c003520 con 0x7f6bb013f710 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 >> v1:172.21.15.132:6790/0 conn(0x7f6bb01430c0 legacy=0x7f6bb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 >> v1:172.21.15.132:6789/0 conn(0x7f6bb00a98f0 legacy=0x7f6bb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 1 monclient: found mon.noname-c 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00b44c0 con 0x7f6bb013f710 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6ba0001690 con 0x7f6bb013f710 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6ba7fff640 10 monclient: dump: 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:46.108 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:46.109 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:46.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6bb7a05640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:19:46.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.105+0000 7f6bb7a05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:46.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.123+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ba0001860 con 0x7f6bb013f710 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.123+0000 7f6ba7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.123+0000 7f6ba7fff640 1 -- 172.21.15.132:0/2386587925 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6ba0001cb0 con 0x7f6bb013f710 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.123+0000 7f6ba7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.123+0000 7f6ba7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.123+0000 7f6ba7fff640 10 monclient: dump: 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:46.125 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:46.126 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 4 set_mon_vals no callback set 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 10 monclient: shutdown 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 1 -- 172.21.15.132:0/2386587925 >> v1:172.21.15.161:6789/0 conn(0x7f6bb013f710 legacy=0x7f6bb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 1 -- 172.21.15.132:0/2386587925 shutdown_connections 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.124+0000 7f6bb7a05640 1 -- 172.21.15.132:0/2386587925 >> 172.21.15.132:0/2386587925 conn(0x7f6bb00b7ea0 msgr2=0x7f6bb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.125+0000 7f6bb7a05640 1 -- 172.21.15.132:0/2386587925 shutdown_connections 2024-03-25T18:19:46.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.125+0000 7f6bb7a05640 1 -- 172.21.15.132:0/2386587925 wait complete. 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.125+0000 7f6bb7a05640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 10 monclient: monmap: 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 1 librados: starting msgr at 2024-03-25T18:19:46.129 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 1 librados: starting objecter 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 1 Processor -- start 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 1 -- start start 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 1 librados: setting wanted keys 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 1 librados: calling monclient init 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.126+0000 7f6bb7a05640 10 monclient: init 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient(hunting): picked mon.noname-c con 0x7f6bb00a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient(hunting): picked mon.noname-b con 0x7f6bb013f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient(hunting): picked mon.noname-a con 0x7f6bb01430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00b07d0 con 0x7f6bb01430c0 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00b0ab0 con 0x7f6bb013f710 2024-03-25T18:19:46.130 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb00af130 con 0x7f6bb00a98f0 2024-03-25T18:19:46.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.127+0000 7f6bb7a05640 10 monclient(hunting): authenticate will time out at 7626.389859s 2024-03-25T18:19:46.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.135+0000 7f6bb57a1640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f6bb00a98f0 0x7f6bb00abb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:46886/0 (socket says 172.21.15.132:46886) 2024-03-25T18:19:46.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.136+0000 7f6bb57a1640 1 -- 172.21.15.132:0/3136072546 learned_addr learned my addr 172.21.15.132:0/3136072546 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:46.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.140+0000 7f6ba5ffb640 10 client.?.objecter ms_handle_connect 0x7f6bb00a98f0 2024-03-25T18:19:46.141 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.140+0000 7f6ba5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.142+0000 7f6ba5ffb640 10 client.?.objecter ms_handle_connect 0x7f6bb013f710 2024-03-25T18:19:46.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.142+0000 7f6ba5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 10 client.?.objecter ms_handle_connect 0x7f6bb01430c0 2024-03-25T18:19:46.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065407904 0 0) 0x7f6bb00af130 con 0x7f6bb00a98f0 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.150+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b94004eb0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.151+0000 7f6bb5fa2640 1 -- 172.21.15.132:0/3136072546 >> v1:172.21.15.132:6789/0 conn(0x7f6bb01430c0 legacy=0x7f6bb01e2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:19:46.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.151+0000 7f6bb5fa2640 1 -- 172.21.15.132:0/3136072546 >> v1:172.21.15.132:6789/0 conn(0x7f6bb01430c0 legacy=0x7f6bb01e2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.151+0000 7f6bb5fa2640 1 --1- 172.21.15.132:0/3136072546 >> v1:172.21.15.132:6789/0 conn(0x7f6bb01430c0 0x7f6bb01e2a50 :-1 s=OPENED pgs=44424 cs=1 l=1).handle_message read tag failed 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.151+0000 7f6bb5fa2640 1 --1- 172.21.15.132:0/3136072546 >> v1:172.21.15.132:6789/0 conn(0x7f6bb01430c0 0x7f6bb01e2a50 :-1 s=OPENED pgs=44424 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.151+0000 7f6ba5ffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.132:6789/0 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.152+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644533600 0 0) 0x7f6bb00b0ab0 con 0x7f6bb013f710 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.152+0000 7f6ba5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.152+0000 7f6ba5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.152+0000 7f6ba5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.153 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.152+0000 7f6ba5ffb640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:19:46.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.152+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bb00af130 con 0x7f6bb013f710 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313260373 0 0) 0x7f6b94004eb0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bb00b0ab0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6bac001960 con 0x7f6bb00a98f0 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.235 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:46.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.233+0000 7f6ba5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:46.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2213390765 0 0) 0x7f6bb00b0ab0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 >> v1:172.21.15.161:6789/0 conn(0x7f6bb013f710 legacy=0x7f6bb00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 >> v1:172.21.15.132:6789/0 conn(0x7f6bb01430c0 legacy=0x7f6bb01e2a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 1 monclient: found mon.c 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb00ac270 con 0x7f6bb00a98f0 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6ba5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 5 monclient: authenticate success, global_id 4496 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 10 monclient: _renew_subs 2024-03-25T18:19:46.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb00acfa0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 10 monclient: _renew_subs 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb00b1420 con 0x7f6bb00a98f0 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.254+0000 7f6bb7a05640 1 librados: init done 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.255+0000 7f6bb7a05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:46.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.256+0000 7f6bb7a05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:46.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.256+0000 7f6bb7a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:19:46.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.256+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb00aef90 con 0x7f6bb00a98f0 2024-03-25T18:19:46.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6bac0035f0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f6bac0037a0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba5ffb640 10 monclient: dump: 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:46.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.584+0000 7f6ba77fe640 4 set_mon_vals no callback set 2024-03-25T18:19:46.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.585+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f6bac0039c0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.586+0000 7f6bb4fa0640 10 monclient: get_auth_request con 0x7f6b9406dd90 auth_method 0 2024-03-25T18:19:46.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.586+0000 7f6ba5ffb640 10 client.4496.objecter ms_handle_connect 0x7f6b9406dd90 2024-03-25T18:19:46.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f6bac0545f0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 10 client.4496.objecter ms_dispatch 0x7f6bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6bac050100 con 0x7f6bb00a98f0 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:46.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.591+0000 7f6ba5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:46.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.838+0000 7f6bb7a05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:46.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.838+0000 7f6bb7a05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:46.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.838+0000 7f6bb7a05640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:19:46.840 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.838+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bb00af2c0 con 0x7f6bb00a98f0 2024-03-25T18:19:46.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.885+0000 7f6ba5ffb640 1 -- 172.21.15.132:0/3136072546 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f6bac052030 con 0x7f6bb00a98f0 2024-03-25T18:19:46.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.885+0000 7f6ba5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:46.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.885+0000 7f6ba5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:46.886 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:46.886 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.887+0000 7f6bb7a05640 10 librados: watch_flush enter 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.887+0000 7f6bb7a05640 10 librados: watch_flush exit 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.887+0000 7f6bb7a05640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.887+0000 7f6bb7a05640 10 client.4496.objecter successfully canceled tick 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.887+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 >> v1:172.21.15.161:6800/411950978 conn(0x7f6b9406dd90 legacy=0x7f6b94070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 10 monclient: shutdown 2024-03-25T18:19:46.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 >> v1:172.21.15.132:6790/0 conn(0x7f6bb00a98f0 legacy=0x7f6bb00abb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 shutdown_connections 2024-03-25T18:19:46.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 >> 172.21.15.132:0/3136072546 conn(0x7f6bb00b7ea0 msgr2=0x7f6bb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 shutdown_connections 2024-03-25T18:19:46.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 1 -- 172.21.15.132:0/3136072546 wait complete. 2024-03-25T18:19:46.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:46.888+0000 7f6bb7a05640 1 librados: shutdown 2024-03-25T18:19:46.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:53.707 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.820+0000 7fed82e2a640 10 monclient: get_monmap_and_config 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.820+0000 7fed82e2a640 10 monclient: build_initial_monmap 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.820+0000 7fed82e2a640 10 monclient: monmap: 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:19:53.821350+0000 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:19:53.821350+0000 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:53.821 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:53.822 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:19:53.822 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:19:53.822 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 1 Processor -- start 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 1 -- start start 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 10 monclient: init 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 10 monclient(hunting): picked mon.noname-c con 0x7fed7c0b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 10 monclient(hunting): picked mon.noname-b con 0x7fed7c0b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 10 monclient(hunting): picked mon.noname-a con 0x7fed7c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.821+0000 7fed82e2a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c0af940 con 0x7fed7c14be90 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.822+0000 7fed82e2a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c0b0a70 con 0x7fed7c0b4d60 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.822+0000 7fed82e2a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c0b0d50 con 0x7fed7c0b1730 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.822+0000 7fed82e2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:53.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.822+0000 7fed82e2a640 10 monclient(hunting): authenticate will time out at 7634.084740s 2024-03-25T18:19:53.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.831+0000 7fed80bc6640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fed7c0b4d60 0x7fed7c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:37852/0 (socket says 172.21.15.132:37852) 2024-03-25T18:19:53.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.831+0000 7fed80bc6640 1 -- 172.21.15.132:0/212773187 learned_addr learned my addr 172.21.15.132:0/212773187 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:53.846 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.845+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361418547 0 0) 0x7fed7c0b0a70 con 0x7fed7c0b4d60 2024-03-25T18:19:53.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.845+0000 7fed7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.845+0000 7fed7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.845+0000 7fed7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.845+0000 7fed7b7fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:19:53.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.845+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6c003520 con 0x7fed7c0b4d60 2024-03-25T18:19:53.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.859+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652588527 0 0) 0x7fed7c0b0d50 con 0x7fed7c0b1730 2024-03-25T18:19:53.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.859+0000 7fed7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.859+0000 7fed7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.859+0000 7fed7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.859+0000 7fed7b7fe640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:19:53.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.859+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7c0b0a70 con 0x7fed7c0b1730 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.862+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1721687973 0 0) 0x7fed6c003520 con 0x7fed7c0b4d60 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 >> v1:172.21.15.132:6790/0 conn(0x7fed7c0b1730 legacy=0x7fed7c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 >> v1:172.21.15.132:6789/0 conn(0x7fed7c14be90 legacy=0x7fed7c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c0b1030 con 0x7fed7c0b4d60 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:53.864 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fed70001690 con 0x7fed7c0b4d60 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed7b7fe640 10 monclient: dump: 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:53.865 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.866 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed82e2a640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:19:53.866 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.863+0000 7fed82e2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed700017f0 con 0x7fed7c0b4d60 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed7b7fe640 1 -- 172.21.15.132:0/212773187 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fed70001c00 con 0x7fed7c0b4d60 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed7b7fe640 10 monclient: dump: 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:53.887 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed82e2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:53.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.886+0000 7fed82e2a640 4 set_mon_vals no callback set 2024-03-25T18:19:53.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 10 monclient: shutdown 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 1 -- 172.21.15.132:0/212773187 >> v1:172.21.15.161:6789/0 conn(0x7fed7c0b4d60 legacy=0x7fed7c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 1 -- 172.21.15.132:0/212773187 shutdown_connections 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 1 -- 172.21.15.132:0/212773187 >> 172.21.15.132:0/212773187 conn(0x7fed7c0617a0 msgr2=0x7fed7c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 1 -- 172.21.15.132:0/212773187 shutdown_connections 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.887+0000 7fed82e2a640 1 -- 172.21.15.132:0/212773187 wait complete. 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 10 monclient: build_initial_monmap 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 10 monclient: monmap: 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:19:53.891 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 1 librados: starting msgr at 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 1 librados: starting objecter 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 1 Processor -- start 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 1 -- start start 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 1 librados: setting wanted keys 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 1 librados: calling monclient init 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.888+0000 7fed82e2a640 10 monclient: init 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:53.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient(hunting): picked mon.noname-a con 0x7fed7c0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient(hunting): picked mon.noname-b con 0x7fed7c0b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient(hunting): picked mon.noname-c con 0x7fed7c14be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c0a9500 con 0x7fed7c0b1730 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c0aa620 con 0x7fed7c0b4d60 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7c0aa900 con 0x7fed7c14be90 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:53.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.889+0000 7fed82e2a640 10 monclient(hunting): authenticate will time out at 7634.152364s 2024-03-25T18:19:53.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.901+0000 7fed80bc6640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fed7c0b1730 0x7fed7c138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:32914/0 (socket says 172.21.15.132:32914) 2024-03-25T18:19:53.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.901+0000 7fed80bc6640 1 -- 172.21.15.132:0/3783852739 learned_addr learned my addr 172.21.15.132:0/3783852739 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:19:53.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.902+0000 7fed797fa640 10 client.?.objecter ms_handle_connect 0x7fed7c0b1730 2024-03-25T18:19:53.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.902+0000 7fed797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:53.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.910+0000 7fed797fa640 10 client.?.objecter ms_handle_connect 0x7fed7c0b4d60 2024-03-25T18:19:53.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.910+0000 7fed797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:53.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.912+0000 7fed797fa640 10 client.?.objecter ms_handle_connect 0x7fed7c14be90 2024-03-25T18:19:53.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.912+0000 7fed797fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:53.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.915+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875262618 0 0) 0x7fed7c0a9500 con 0x7fed7c0b1730 2024-03-25T18:19:53.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.915+0000 7fed797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.915+0000 7fed797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.915+0000 7fed797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.915+0000 7fed797fa640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:19:53.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.915+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed60004eb0 con 0x7fed7c0b1730 2024-03-25T18:19:53.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.923+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198252158 0 0) 0x7fed7c0aa620 con 0x7fed7c0b4d60 2024-03-25T18:19:53.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.923+0000 7fed797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.923+0000 7fed797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.923+0000 7fed797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.923+0000 7fed797fa640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:19:53.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.923+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7c0a9500 con 0x7fed7c0b4d60 2024-03-25T18:19:53.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.925+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157043734 0 0) 0x7fed7c0aa900 con 0x7fed7c14be90 2024-03-25T18:19:53.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.926+0000 7fed797fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:53.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.926+0000 7fed797fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:53.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.926+0000 7fed797fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:53.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.926+0000 7fed797fa640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:19:53.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.926+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7c0aa620 con 0x7fed7c14be90 2024-03-25T18:19:53.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 361431549 0 0) 0x7fed60004eb0 con 0x7fed7c0b1730 2024-03-25T18:19:53.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed7c0aa900 con 0x7fed7c0b1730 2024-03-25T18:19:53.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fed70001b40 con 0x7fed7c0b1730 2024-03-25T18:19:53.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 10 monclient(hunting): dump: 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.928+0000 7fed797fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:53.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.932+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2189290293 0 0) 0x7fed7c0a9500 con 0x7fed7c0b4d60 2024-03-25T18:19:53.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.932+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed60004eb0 con 0x7fed7c0b4d60 2024-03-25T18:19:53.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.932+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fed68001930 con 0x7fed7c0b4d60 2024-03-25T18:19:53.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.932+0000 7fed797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.932+0000 7fed797fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.932+0000 7fed797fa640 10 monclient(hunting): dump: 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:53.934 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.950+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507498640 0 0) 0x7fed7c0aa620 con 0x7fed7c14be90 2024-03-25T18:19:53.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.950+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed7c0a9500 con 0x7fed7c14be90 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.950+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fed74001a10 con 0x7fed7c14be90 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.950+0000 7fed797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.950+0000 7fed797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.950+0000 7fed797fa640 10 monclient(hunting): dump: 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:53.952 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:53.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3691435941 0 0) 0x7fed7c0aa900 con 0x7fed7c0b1730 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 >> v1:172.21.15.132:6790/0 conn(0x7fed7c14be90 legacy=0x7fed7c1db4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 >> v1:172.21.15.161:6789/0 conn(0x7fed7c0b4d60 legacy=0x7fed7c1dad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 1 monclient: found mon.a 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c138820 con 0x7fed7c0b1730 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:19:53.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 10 monclient: _renew_subs 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c142e30 con 0x7fed7c0b1730 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 10 client.4506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 10 monclient: _renew_subs 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c142bc0 con 0x7fed7c0b1730 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.956+0000 7fed82e2a640 1 librados: init done 2024-03-25T18:19:53.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.957+0000 7fed82e2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:53.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.957+0000 7fed82e2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:53.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.957+0000 7fed82e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:19:53.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:53.957+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7c0627b0 con 0x7fed7c0b1730 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed70004a90 con 0x7fed7c0b1730 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fed70004da0 con 0x7fed7c0b1730 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 10 monclient: dump: 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:19:54.272 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:19:54.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed7affd640 4 set_mon_vals no callback set 2024-03-25T18:19:54.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.271+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fed7001ad70 con 0x7fed7c0b1730 2024-03-25T18:19:54.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.272+0000 7fed7bfff640 10 monclient: get_auth_request con 0x7fed6006e0f0 auth_method 0 2024-03-25T18:19:54.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.272+0000 7fed797fa640 10 client.4506.objecter ms_handle_connect 0x7fed6006e0f0 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fed700548d0 con 0x7fed7c0b1730 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 10 client.4506.objecter ms_dispatch 0x7fed7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 3 client.4506.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 3 client.4506.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 20 client.4506.objecter dump_active .. 0 homeless 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed700503e0 con 0x7fed7c0b1730 2024-03-25T18:19:54.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.279+0000 7fed797fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:54.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.518+0000 7fed82e2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:54.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.518+0000 7fed82e2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:54.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.518+0000 7fed82e2a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:19:54.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.518+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed7c13cba0 con 0x7fed7c0b1730 2024-03-25T18:19:54.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.556+0000 7fed797fa640 1 -- 172.21.15.132:0/3783852739 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fed70052310 con 0x7fed7c0b1730 2024-03-25T18:19:54.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.556+0000 7fed797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:54.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.556+0000 7fed797fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:54.558 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:19:54.558 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:54.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.558+0000 7fed82e2a640 10 librados: watch_flush enter 2024-03-25T18:19:54.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.558+0000 7fed82e2a640 10 librados: watch_flush exit 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.558+0000 7fed82e2a640 20 client.4506.objecter shutdown clearing up homeless session... 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.558+0000 7fed82e2a640 10 client.4506.objecter successfully canceled tick 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.558+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 >> v1:172.21.15.161:6800/411950978 conn(0x7fed6006e0f0 legacy=0x7fed600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.558+0000 7fed82e2a640 10 monclient: shutdown 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 >> v1:172.21.15.132:6789/0 conn(0x7fed7c0b1730 legacy=0x7fed7c138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 shutdown_connections 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 >> 172.21.15.132:0/3783852739 conn(0x7fed7c0617a0 msgr2=0x7fed7c13c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 shutdown_connections 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 1 -- 172.21.15.132:0/3783852739 wait complete. 2024-03-25T18:19:54.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:19:54.559+0000 7fed82e2a640 1 librados: shutdown 2024-03-25T18:19:54.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:55.078 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:40.342 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.089 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:40.353 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.133 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:06:40.397 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.139 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:40.403 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.147 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:40.389 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.152 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:40.393 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.152 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:40.416 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.178 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:06:40.420 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:55.181 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:06:40.445 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:01.377 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.493+0000 7f154153d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.493+0000 7f154153d640 10 monclient: build_initial_monmap 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.493+0000 7f154153d640 10 monclient: monmap: 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:01.494911+0000 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:01.494911+0000 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:01.495 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:01.496 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:01.496 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:01.496 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.494+0000 7f154153d640 1 Processor -- start 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.494+0000 7f154153d640 1 -- start start 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.494+0000 7f154153d640 10 monclient: init 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient(hunting): picked mon.noname-b con 0x7f153c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient(hunting): picked mon.noname-c con 0x7f153c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient(hunting): picked mon.noname-a con 0x7f153c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c142520 con 0x7f153c1430c0 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0b28a0 con 0x7f153c13f710 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c0b2a30 con 0x7f153c0a98f0 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:01.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.495+0000 7f154153d640 10 monclient(hunting): authenticate will time out at 7641.758350s 2024-03-25T18:20:01.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.511+0000 7f153affd640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f153c0a98f0 0x7f153c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:52276/0 (socket says 172.21.15.132:52276) 2024-03-25T18:20:01.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.511+0000 7f153affd640 1 -- 172.21.15.132:0/1772491696 learned_addr learned my addr 172.21.15.132:0/1772491696 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:01.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.520+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441817961 0 0) 0x7f153c0b2a30 con 0x7f153c0a98f0 2024-03-25T18:20:01.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.520+0000 7f153a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.520+0000 7f153a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.520+0000 7f153a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.520+0000 7f153a7fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:20:01.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.520+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1528003520 con 0x7f153c0a98f0 2024-03-25T18:20:01.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3218706089 0 0) 0x7f1528003520 con 0x7f153c0a98f0 2024-03-25T18:20:01.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 >> v1:172.21.15.161:6789/0 conn(0x7f153c13f710 legacy=0x7f153c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 >> v1:172.21.15.132:6789/0 conn(0x7f153c1430c0 legacy=0x7f153c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:01.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0b44c0 con 0x7f153c0a98f0 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f152c001690 con 0x7f153c0a98f0 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f153a7fc640 10 monclient: dump: 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.530 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f154153d640 5 monclient: authenticate success, global_id 4505 2024-03-25T18:20:01.531 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.528+0000 7f154153d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.538+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f152c0017f0 con 0x7f153c0a98f0 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.538+0000 7f153a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.538+0000 7f153a7fc640 1 -- 172.21.15.132:0/1772491696 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f152c001c00 con 0x7f153c0a98f0 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.538+0000 7f153a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.538+0000 7f153a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.538+0000 7f153a7fc640 10 monclient: dump: 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:01.540 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:01.541 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:01.541 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:01.541 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:01.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.539+0000 7f154153d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:01.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.539+0000 7f154153d640 4 set_mon_vals no callback set 2024-03-25T18:20:01.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.539+0000 7f154153d640 10 monclient: shutdown 2024-03-25T18:20:01.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.539+0000 7f154153d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:01.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.539+0000 7f154153d640 1 -- 172.21.15.132:0/1772491696 >> v1:172.21.15.132:6790/0 conn(0x7f153c0a98f0 legacy=0x7f153c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.540+0000 7f154153d640 1 -- 172.21.15.132:0/1772491696 shutdown_connections 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.540+0000 7f154153d640 1 -- 172.21.15.132:0/1772491696 >> 172.21.15.132:0/1772491696 conn(0x7f153c0b7ea0 msgr2=0x7f153c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.540+0000 7f154153d640 1 -- 172.21.15.132:0/1772491696 shutdown_connections 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.540+0000 7f154153d640 1 -- 172.21.15.132:0/1772491696 wait complete. 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 10 monclient: build_initial_monmap 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 10 monclient: monmap: 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:01.543 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 1 librados: starting msgr at 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 1 librados: starting objecter 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 1 Processor -- start 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 1 -- start start 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 1 librados: setting wanted keys 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 1 librados: calling monclient init 2024-03-25T18:20:01.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.541+0000 7f154153d640 10 monclient: init 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient(hunting): picked mon.noname-c con 0x7f153c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient(hunting): picked mon.noname-b con 0x7f153c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient(hunting): picked mon.noname-a con 0x7f153c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c135730 con 0x7f153c1430c0 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c1358c0 con 0x7f153c13f710 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c135a50 con 0x7f153c0a98f0 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:01.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.542+0000 7f154153d640 10 monclient(hunting): authenticate will time out at 7641.805210s 2024-03-25T18:20:01.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.550+0000 7f1533fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f153c13f710 0x7f153c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:42420/0 (socket says 172.21.15.132:42420) 2024-03-25T18:20:01.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.550+0000 7f1533fff640 1 -- 172.21.15.132:0/3768540969 learned_addr learned my addr 172.21.15.132:0/3768540969 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:01.554 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.553+0000 7f15337fe640 10 client.?.objecter ms_handle_connect 0x7f153c0a98f0 2024-03-25T18:20:01.554 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.553+0000 7f15337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:01.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.555+0000 7f15337fe640 10 client.?.objecter ms_handle_connect 0x7f153c1430c0 2024-03-25T18:20:01.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.555+0000 7f15337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:01.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.555+0000 7f15337fe640 10 client.?.objecter ms_handle_connect 0x7f153c13f710 2024-03-25T18:20:01.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.555+0000 7f15337fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:01.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.559+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484909617 0 0) 0x7f153c135730 con 0x7f153c1430c0 2024-03-25T18:20:01.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.559+0000 7f15337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.559+0000 7f15337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.559+0000 7f15337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.559+0000 7f15337fe640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:20:01.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.559+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c004eb0 con 0x7f153c1430c0 2024-03-25T18:20:01.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.562+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3835799076 0 0) 0x7f153c135a50 con 0x7f153c0a98f0 2024-03-25T18:20:01.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.562+0000 7f15337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.562+0000 7f15337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.562+0000 7f15337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.562+0000 7f15337fe640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:20:01.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.562+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c135730 con 0x7f153c0a98f0 2024-03-25T18:20:01.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.567+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262417322 0 0) 0x7f153c1358c0 con 0x7f153c13f710 2024-03-25T18:20:01.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.567+0000 7f15337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:01.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.567+0000 7f15337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:01.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.567+0000 7f15337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:01.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.567+0000 7f15337fe640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:20:01.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.567+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f153c135a50 con 0x7f153c13f710 2024-03-25T18:20:01.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4240537463 0 0) 0x7f151c004eb0 con 0x7f153c1430c0 2024-03-25T18:20:01.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f153c1358c0 con 0x7f153c1430c0 2024-03-25T18:20:01.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1534001b90 con 0x7f153c1430c0 2024-03-25T18:20:01.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 10 monclient(hunting): dump: 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:01.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.569+0000 7f15337fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:01.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3198805899 0 0) 0x7f153c1358c0 con 0x7f153c1430c0 2024-03-25T18:20:01.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 >> v1:172.21.15.132:6790/0 conn(0x7f153c0a98f0 legacy=0x7f153c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 >> v1:172.21.15.161:6789/0 conn(0x7f153c13f710 legacy=0x7f153c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:01.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 1 monclient: found mon.a 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c1d5000 con 0x7f153c1430c0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f15337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 5 monclient: authenticate success, global_id 4512 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 10 monclient: _renew_subs 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1d52b0 con 0x7f153c1430c0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 10 monclient: _renew_subs 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:01.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c13c900 con 0x7f153c1430c0 2024-03-25T18:20:01.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.579+0000 7f154153d640 1 librados: init done 2024-03-25T18:20:01.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.580+0000 7f154153d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:01.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.580+0000 7f154153d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:01.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.580+0000 7f154153d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:01.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.580+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c13cbb0 con 0x7f153c1430c0 2024-03-25T18:20:01.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.582+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1534002080 con 0x7f153c1430c0 2024-03-25T18:20:01.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.582+0000 7f15337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:01.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.582+0000 7f1539ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.878+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1534003990 con 0x7f153c1430c0 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.878+0000 7f15337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.878+0000 7f15337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.878+0000 7f15337fe640 10 monclient: dump: 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:01.879 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:01.880 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:01.880 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:01.880 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:01.880 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:01.880 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:01.880 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.878+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f1534003bb0 con 0x7f153c1430c0 2024-03-25T18:20:01.881 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.879+0000 7f153affd640 10 monclient: get_auth_request con 0x7f151c06dd50 auth_method 0 2024-03-25T18:20:01.881 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.879+0000 7f15337fe640 10 client.4512.objecter ms_handle_connect 0x7f151c06dd50 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f15340537a0 con 0x7f153c1430c0 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 10 client.4512.objecter ms_dispatch 0x7f153c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f153404f0e0 con 0x7f153c1430c0 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:01.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:01.885+0000 7f15337fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:02.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.135+0000 7f154153d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.135+0000 7f154153d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.135+0000 7f154153d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:02.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.135+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f153c1371c0 con 0x7f153c1430c0 2024-03-25T18:20:02.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.166+0000 7f15337fe640 1 -- 172.21.15.132:0/3768540969 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f15340511e0 con 0x7f153c1430c0 2024-03-25T18:20:02.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.166+0000 7f15337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.166+0000 7f15337fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:02.167 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:02.167 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:02.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 10 librados: watch_flush enter 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 10 librados: watch_flush exit 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 10 client.4512.objecter successfully canceled tick 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 >> v1:172.21.15.161:6800/411950978 conn(0x7f151c06dd50 legacy=0x7f151c070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 10 monclient: shutdown 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 >> v1:172.21.15.132:6789/0 conn(0x7f153c1430c0 legacy=0x7f153c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 shutdown_connections 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 >> 172.21.15.132:0/3768540969 conn(0x7f153c0b7ea0 msgr2=0x7f153c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 shutdown_connections 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 -- 172.21.15.132:0/3768540969 wait complete. 2024-03-25T18:20:02.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:02.168+0000 7f154153d640 1 librados: shutdown 2024-03-25T18:20:02.180 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:08.986 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:09.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.098+0000 7fa239a86640 10 monclient: get_monmap_and_config 2024-03-25T18:20:09.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.098+0000 7fa239a86640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.098+0000 7fa239a86640 10 monclient: monmap: 2024-03-25T18:20:09.099 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:09.099 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:09.099 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:09.099383+0000 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:09.099383+0000 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:09.100 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 1 Processor -- start 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 1 -- start start 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient: init 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient(hunting): picked mon.noname-c con 0x7fa2340a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient(hunting): picked mon.noname-b con 0x7fa23413f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:09.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient(hunting): picked mon.noname-a con 0x7fa2341430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:09.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa234142520 con 0x7fa2341430c0 2024-03-25T18:20:09.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2340b28a0 con 0x7fa2340a98f0 2024-03-25T18:20:09.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2340b2a30 con 0x7fa23413f710 2024-03-25T18:20:09.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.099+0000 7fa239a86640 10 monclient(hunting): authenticate will time out at 7649.362657s 2024-03-25T18:20:09.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.117+0000 7fa2337fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa2340a98f0 0x7fa2340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:50222/0 (socket says 172.21.15.132:50222) 2024-03-25T18:20:09.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.117+0000 7fa2337fe640 1 -- 172.21.15.132:0/509652350 learned_addr learned my addr 172.21.15.132:0/509652350 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.131+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595649526 0 0) 0x7fa2340b28a0 con 0x7fa2340a98f0 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.131+0000 7fa2327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.131+0000 7fa2327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.131+0000 7fa2327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.131+0000 7fa2327fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:20:09.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.131+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c003520 con 0x7fa2340a98f0 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.134+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724816912 0 0) 0x7fa2340b2a30 con 0x7fa23413f710 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.134+0000 7fa2327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.134+0000 7fa2327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.134+0000 7fa2327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.134+0000 7fa2327fc640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:20:09.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.134+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2340b28a0 con 0x7fa23413f710 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2163760171 0 0) 0x7fa2340b28a0 con 0x7fa23413f710 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 >> v1:172.21.15.161:6789/0 conn(0x7fa2340a98f0 legacy=0x7fa2340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 >> v1:172.21.15.132:6789/0 conn(0x7fa2341430c0 legacy=0x7fa23414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2340b44c0 con 0x7fa23413f710 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa228001670 con 0x7fa23413f710 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa239a86640 5 monclient: authenticate success, global_id 4511 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa239a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa2327fc640 10 monclient: dump: 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:09.147 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:09.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.144+0000 7fa239a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:09.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2280017d0 con 0x7fa23413f710 2024-03-25T18:20:09.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa2327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa2327fc640 1 -- 172.21.15.132:0/509652350 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa228001ba0 con 0x7fa23413f710 2024-03-25T18:20:09.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa2327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa2327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa2327fc640 10 monclient: dump: 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa239a86640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:09.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.165+0000 7fa239a86640 4 set_mon_vals no callback set 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 10 monclient: shutdown 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 1 -- 172.21.15.132:0/509652350 >> v1:172.21.15.132:6790/0 conn(0x7fa23413f710 legacy=0x7fa2341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 1 -- 172.21.15.132:0/509652350 shutdown_connections 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 1 -- 172.21.15.132:0/509652350 >> 172.21.15.132:0/509652350 conn(0x7fa2340b7ea0 msgr2=0x7fa2340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 1 -- 172.21.15.132:0/509652350 shutdown_connections 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.166+0000 7fa239a86640 1 -- 172.21.15.132:0/509652350 wait complete. 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 10 monclient: build_initial_monmap 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 10 monclient: monmap: 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:09.170 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 1 librados: starting msgr at 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 1 librados: starting objecter 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 1 Processor -- start 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 1 -- start start 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 1 librados: setting wanted keys 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 1 librados: calling monclient init 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.167+0000 7fa239a86640 10 monclient: init 2024-03-25T18:20:09.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient(hunting): picked mon.noname-c con 0x7fa2340a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient(hunting): picked mon.noname-a con 0x7fa23413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient(hunting): picked mon.noname-b con 0x7fa2341430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23413b720 con 0x7fa23413f710 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa234139d20 con 0x7fa2341430c0 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2340b7b70 con 0x7fa2340a98f0 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.168+0000 7fa239a86640 10 monclient(hunting): authenticate will time out at 7649.431158s 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.169+0000 7fa232ffd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa23413f710 0x7fa2341df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:50748/0 (socket says 172.21.15.132:50748) 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.169+0000 7fa232ffd640 1 -- 172.21.15.132:0/1230835893 learned_addr learned my addr 172.21.15.132:0/1230835893 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.169+0000 7fa21bfff640 10 client.?.objecter ms_handle_connect 0x7fa23413f710 2024-03-25T18:20:09.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.169+0000 7fa21bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.170+0000 7fa21bfff640 10 client.?.objecter ms_handle_connect 0x7fa2341430c0 2024-03-25T18:20:09.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.170+0000 7fa21bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.235+0000 7fa21bfff640 10 client.?.objecter ms_handle_connect 0x7fa2340a98f0 2024-03-25T18:20:09.236 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.235+0000 7fa21bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:09.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.250+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74517576 0 0) 0x7fa2340b7b70 con 0x7fa2340a98f0 2024-03-25T18:20:09.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.250+0000 7fa21bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.250+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.250+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.250+0000 7fa21bfff640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:20:09.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.250+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa220004eb0 con 0x7fa2340a98f0 2024-03-25T18:20:09.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.291+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130244077 0 0) 0x7fa234139d20 con 0x7fa2341430c0 2024-03-25T18:20:09.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.291+0000 7fa21bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.291+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.291+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.291+0000 7fa21bfff640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:20:09.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.291+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2340b7b70 con 0x7fa2341430c0 2024-03-25T18:20:09.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3323442238 0 0) 0x7fa220004eb0 con 0x7fa2340a98f0 2024-03-25T18:20:09.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa234139d20 con 0x7fa2340a98f0 2024-03-25T18:20:09.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa224001950 con 0x7fa2340a98f0 2024-03-25T18:20:09.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:09.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 10 monclient(hunting): dump: 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:09.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.292+0000 7fa21bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:09.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.299+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659683351 0 0) 0x7fa23413b720 con 0x7fa23413f710 2024-03-25T18:20:09.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.299+0000 7fa21bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:09.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.299+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:09.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.299+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:09.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.299+0000 7fa21bfff640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:20:09.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.299+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa220004eb0 con 0x7fa23413f710 2024-03-25T18:20:09.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.314+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2310756261 0 0) 0x7fa234139d20 con 0x7fa2340a98f0 2024-03-25T18:20:09.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.314+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 >> v1:172.21.15.161:6789/0 conn(0x7fa2341430c0 legacy=0x7fa2341e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.314+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 >> v1:172.21.15.132:6789/0 conn(0x7fa23413f710 legacy=0x7fa2341df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:09.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 1 monclient: found mon.c 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23413c250 con 0x7fa2340a98f0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 10 monclient: _finish_auth 0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa21bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 10 monclient: _renew_subs 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2340b71f0 con 0x7fa2340a98f0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 10 monclient: _renew_subs 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa23412f910 con 0x7fa2340a98f0 2024-03-25T18:20:09.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.315+0000 7fa239a86640 1 librados: init done 2024-03-25T18:20:09.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.316+0000 7fa239a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.316+0000 7fa239a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.316+0000 7fa239a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:09.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.316+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa234139eb0 con 0x7fa2340a98f0 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2240047e0 con 0x7fa2340a98f0 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa21bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa224001d80 con 0x7fa2340a98f0 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa21bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa21bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa21bfff640 10 monclient: dump: 2024-03-25T18:20:09.351 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.349+0000 7fa231ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:09.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.350+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fa224004c30 con 0x7fa2340a98f0 2024-03-25T18:20:09.353 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.351+0000 7fa232ffd640 10 monclient: get_auth_request con 0x7fa22006de00 auth_method 0 2024-03-25T18:20:09.353 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.351+0000 7fa21bfff640 10 client.4514.objecter ms_handle_connect 0x7fa22006de00 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fa224054850 con 0x7fa2340a98f0 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 10 client.4514.objecter ms_dispatch 0x7fa234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa224050190 con 0x7fa2340a98f0 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:09.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.651+0000 7fa21bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:09.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.906+0000 7fa239a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:09.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.906+0000 7fa239a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.906+0000 7fa239a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:09.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.906+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa23413c3e0 con 0x7fa2340a98f0 2024-03-25T18:20:09.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.953+0000 7fa21bfff640 1 -- 172.21.15.132:0/1230835893 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fa224052290 con 0x7fa2340a98f0 2024-03-25T18:20:09.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.954+0000 7fa21bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:09.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.954+0000 7fa21bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:09.955 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:09.955 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 10 librados: watch_flush enter 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 10 librados: watch_flush exit 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 10 client.4514.objecter successfully canceled tick 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 >> v1:172.21.15.161:6800/411950978 conn(0x7fa22006de00 legacy=0x7fa2200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 10 monclient: shutdown 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 >> v1:172.21.15.132:6790/0 conn(0x7fa2340a98f0 legacy=0x7fa23413bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 shutdown_connections 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 >> 172.21.15.132:0/1230835893 conn(0x7fa2340b7ea0 msgr2=0x7fa234074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 shutdown_connections 2024-03-25T18:20:09.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 -- 172.21.15.132:0/1230835893 wait complete. 2024-03-25T18:20:09.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:09.956+0000 7fa239a86640 1 librados: shutdown 2024-03-25T18:20:09.969 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:15.163 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:00.426 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.167 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:00.431 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.198 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:00.462 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.236 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:00.477 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.240 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:00.481 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.244 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:00.508 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.249 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:00.513 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.278 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:00.519 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:15.284 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:00.549 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:16.777 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:16.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.891+0000 7f1b25488640 10 monclient: get_monmap_and_config 2024-03-25T18:20:16.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.891+0000 7f1b25488640 10 monclient: build_initial_monmap 2024-03-25T18:20:16.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.891+0000 7f1b25488640 10 monclient: monmap: 2024-03-25T18:20:16.891 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:16.891 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:16.891249+0000 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:16.891249+0000 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:16.892 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:16.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.891+0000 7f1b25488640 1 Processor -- start 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 1 -- start start 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient: init 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient(hunting): picked mon.noname-a con 0x7f1b200a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient(hunting): picked mon.noname-b con 0x7f1b2013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient(hunting): picked mon.noname-c con 0x7f1b201430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b20142520 con 0x7f1b200a98f0 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200b28a0 con 0x7f1b201430c0 2024-03-25T18:20:16.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200b2a30 con 0x7f1b2013f710 2024-03-25T18:20:16.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:16.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.892+0000 7f1b25488640 10 monclient(hunting): authenticate will time out at 7657.154411s 2024-03-25T18:20:16.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.903+0000 7f1b17fff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f1b2013f710 0x7f1b201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:45018/0 (socket says 172.21.15.132:45018) 2024-03-25T18:20:16.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.903+0000 7f1b17fff640 1 -- 172.21.15.132:0/1578253834 learned_addr learned my addr 172.21.15.132:0/1578253834 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:16.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.918+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914526984 0 0) 0x7f1b200b2a30 con 0x7f1b2013f710 2024-03-25T18:20:16.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.918+0000 7f1b1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.918+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.918+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.918+0000 7f1b1e7fc640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:20:16.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.918+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c003520 con 0x7f1b2013f710 2024-03-25T18:20:16.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.927+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995214902 0 0) 0x7f1b200b28a0 con 0x7f1b201430c0 2024-03-25T18:20:16.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.927+0000 7f1b1e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.927+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.927+0000 7f1b1e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.927+0000 7f1b1e7fc640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:20:16.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.927+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b200b2a30 con 0x7f1b201430c0 2024-03-25T18:20:16.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881921775 0 0) 0x7f1b0c003520 con 0x7f1b2013f710 2024-03-25T18:20:16.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 >> v1:172.21.15.161:6789/0 conn(0x7f1b201430c0 legacy=0x7f1b2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 >> v1:172.21.15.132:6789/0 conn(0x7f1b200a98f0 legacy=0x7f1b200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200b44c0 con 0x7f1b2013f710 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b08001690 con 0x7f1b2013f710 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b25488640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b25488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:16.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b1e7fc640 10 monclient: dump: 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:16.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.931+0000 7f1b25488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:16.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b080017f0 con 0x7f1b2013f710 2024-03-25T18:20:16.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b1e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:16.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b1e7fc640 1 -- 172.21.15.132:0/1578253834 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b08001c00 con 0x7f1b2013f710 2024-03-25T18:20:16.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b1e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:16.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b1e7fc640 10 monclient: dump: 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b25488640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:16.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b25488640 4 set_mon_vals no callback set 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b25488640 10 monclient: shutdown 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b25488640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b25488640 1 -- 172.21.15.132:0/1578253834 >> v1:172.21.15.132:6790/0 conn(0x7f1b2013f710 legacy=0x7f1b201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.945+0000 7f1b25488640 1 -- 172.21.15.132:0/1578253834 shutdown_connections 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 1 -- 172.21.15.132:0/1578253834 >> 172.21.15.132:0/1578253834 conn(0x7f1b200b7ea0 msgr2=0x7f1b200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 1 -- 172.21.15.132:0/1578253834 shutdown_connections 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 1 -- 172.21.15.132:0/1578253834 wait complete. 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 10 monclient: build_initial_monmap 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 10 monclient: monmap: 2024-03-25T18:20:16.949 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 1 librados: starting msgr at 2024-03-25T18:20:16.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.946+0000 7f1b25488640 1 librados: starting objecter 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 1 Processor -- start 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 1 -- start start 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 1 librados: setting wanted keys 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 1 librados: calling monclient init 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 10 monclient: init 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 10 monclient(hunting): picked mon.noname-a con 0x7f1b200a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 10 monclient(hunting): picked mon.noname-c con 0x7f1b2013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 10 monclient(hunting): picked mon.noname-b con 0x7f1b201430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200b1870 con 0x7f1b200a98f0 2024-03-25T18:20:16.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.947+0000 7f1b25488640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200ac910 con 0x7f1b201430c0 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.948+0000 7f1b25488640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b200abb00 con 0x7f1b2013f710 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.948+0000 7f1b25488640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:16.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.948+0000 7f1b25488640 10 monclient(hunting): authenticate will time out at 7657.209743s 2024-03-25T18:20:16.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.960+0000 7f1b1effd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f1b200a98f0 0x7f1b20138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:41902/0 (socket says 172.21.15.132:41902) 2024-03-25T18:20:16.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.960+0000 7f1b1effd640 1 -- 172.21.15.132:0/3126476845 learned_addr learned my addr 172.21.15.132:0/3126476845 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:16.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.962+0000 7f1b177fe640 10 client.?.objecter ms_handle_connect 0x7f1b200a98f0 2024-03-25T18:20:16.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.962+0000 7f1b177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749899010 0 0) 0x7f1b20142520 con 0x7f1b200a98f0 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b00004eb0 con 0x7f1b200a98f0 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 10 client.?.objecter ms_handle_connect 0x7f1b2013f710 2024-03-25T18:20:16.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.968+0000 7f1b177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:16.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.972+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153821213 0 0) 0x7f1b200abb00 con 0x7f1b2013f710 2024-03-25T18:20:16.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.972+0000 7f1b177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:16.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.972+0000 7f1b177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:16.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.972+0000 7f1b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:16.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.972+0000 7f1b177fe640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:20:16.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.972+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b20142520 con 0x7f1b2013f710 2024-03-25T18:20:16.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.978+0000 7f1b177fe640 10 client.?.objecter ms_handle_connect 0x7f1b201430c0 2024-03-25T18:20:16.979 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.978+0000 7f1b177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2824165365 0 0) 0x7f1b00004eb0 con 0x7f1b200a98f0 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b200abb00 con 0x7f1b200a98f0 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b10001890 con 0x7f1b200a98f0 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 10 monclient(hunting): dump: 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:16.983 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:16.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.983+0000 7f1b177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:16.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 17813823 0 0) 0x7f1b200abb00 con 0x7f1b200a98f0 2024-03-25T18:20:16.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 >> v1:172.21.15.132:6790/0 conn(0x7f1b2013f710 legacy=0x7f1b200ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 >> v1:172.21.15.161:6789/0 conn(0x7f1b201430c0 legacy=0x7f1b201e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:16.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 1 monclient: found mon.a 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b200abde0 con 0x7f1b200a98f0 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b25488640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b25488640 10 monclient: _renew_subs 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b25488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.989+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b20138f40 con 0x7f1b200a98f0 2024-03-25T18:20:16.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.990+0000 7f1b25488640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:16.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.990+0000 7f1b25488640 10 monclient: _renew_subs 2024-03-25T18:20:16.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.990+0000 7f1b25488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:16.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.990+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b201363f0 con 0x7f1b200a98f0 2024-03-25T18:20:16.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.990+0000 7f1b25488640 1 librados: init done 2024-03-25T18:20:16.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.991+0000 7f1b25488640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:16.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.991+0000 7f1b25488640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:16.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.991+0000 7f1b25488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:16.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:16.991+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b200627b0 con 0x7f1b200a98f0 2024-03-25T18:20:17.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b10003610 con 0x7f1b200a98f0 2024-03-25T18:20:17.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b10001d80 con 0x7f1b200a98f0 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 10 monclient: dump: 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:17.302 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:17.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b1dffb640 4 set_mon_vals no callback set 2024-03-25T18:20:17.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.301+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f1b10003aa0 con 0x7f1b200a98f0 2024-03-25T18:20:17.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.302+0000 7f1b17fff640 10 monclient: get_auth_request con 0x7f1b0006dda0 auth_method 0 2024-03-25T18:20:17.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.302+0000 7f1b177fe640 10 client.4521.objecter ms_handle_connect 0x7f1b0006dda0 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.309+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f1b100546e0 con 0x7f1b200a98f0 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.309+0000 7f1b177fe640 10 client.4521.objecter ms_dispatch 0x7f1b20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.309+0000 7f1b177fe640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:17.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.309+0000 7f1b177fe640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.310+0000 7f1b177fe640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.310+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b100501f0 con 0x7f1b200a98f0 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.310+0000 7f1b177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.310+0000 7f1b177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:17.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.576+0000 7f1b25488640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:17.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.576+0000 7f1b25488640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:17.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.576+0000 7f1b25488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:20:17.576 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.576+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b201434a0 con 0x7f1b200a98f0 2024-03-25T18:20:17.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.602+0000 7f1b177fe640 1 -- 172.21.15.132:0/3126476845 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f1b10052120 con 0x7f1b200a98f0 2024-03-25T18:20:17.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.602+0000 7f1b177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:17.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.602+0000 7f1b177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:17.602 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:17.603 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:17.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 10 librados: watch_flush enter 2024-03-25T18:20:17.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 10 librados: watch_flush exit 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 >> v1:172.21.15.161:6800/411950978 conn(0x7f1b0006dda0 legacy=0x7f1b00070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 10 monclient: shutdown 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 >> v1:172.21.15.132:6789/0 conn(0x7f1b200a98f0 legacy=0x7f1b20138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 shutdown_connections 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 >> 172.21.15.132:0/3126476845 conn(0x7f1b200b7ea0 msgr2=0x7f1b20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 shutdown_connections 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.604+0000 7f1b25488640 1 -- 172.21.15.132:0/3126476845 wait complete. 2024-03-25T18:20:17.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:17.605+0000 7f1b25488640 1 librados: shutdown 2024-03-25T18:20:17.617 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:24.423 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:24.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.533+0000 7fbfb3f3b640 10 monclient: get_monmap_and_config 2024-03-25T18:20:24.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.533+0000 7fbfb3f3b640 10 monclient: build_initial_monmap 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.533+0000 7fbfb3f3b640 10 monclient: monmap: 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:24.534662+0000 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:24.534662+0000 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:24.535 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.534+0000 7fbfb3f3b640 1 Processor -- start 2024-03-25T18:20:24.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.534+0000 7fbfb3f3b640 1 -- start start 2024-03-25T18:20:24.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.534+0000 7fbfb3f3b640 10 monclient: init 2024-03-25T18:20:24.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:24.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient(hunting): picked mon.noname-a con 0x7fbfac0b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient(hunting): picked mon.noname-b con 0x7fbfac0b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient(hunting): picked mon.noname-c con 0x7fbfac14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0af940 con 0x7fbfac0b4d60 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0b0a70 con 0x7fbfac14be90 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac0b0d50 con 0x7fbfac0b1730 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:24.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.535+0000 7fbfb3f3b640 10 monclient(hunting): authenticate will time out at 7664.797915s 2024-03-25T18:20:24.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.543+0000 7fbfb14d6640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fbfac0b1730 0x7fbfac0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:54850/0 (socket says 172.21.15.132:54850) 2024-03-25T18:20:24.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.543+0000 7fbfb14d6640 1 -- 172.21.15.132:0/938996997 learned_addr learned my addr 172.21.15.132:0/938996997 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127067407 0 0) 0x7fbfac0b0a70 con 0x7fbfac14be90 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf98003520 con 0x7fbfac14be90 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 311799793 0 0) 0x7fbfac0af940 con 0x7fbfac0b4d60 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.562 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac0b0a70 con 0x7fbfac0b4d60 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633837300 0 0) 0x7fbfac0b0d50 con 0x7fbfac0b1730 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:20:24.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.560+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac0af940 con 0x7fbfac0b1730 2024-03-25T18:20:24.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4174370239 0 0) 0x7fbfac0b0a70 con 0x7fbfac0b4d60 2024-03-25T18:20:24.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 >> v1:172.21.15.132:6790/0 conn(0x7fbfac0b1730 legacy=0x7fbfac0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 >> v1:172.21.15.161:6789/0 conn(0x7fbfac14be90 legacy=0x7fbfac14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: _finish_hunting 0 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 1 monclient: found mon.noname-a 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0b1030 con 0x7fbfac0b4d60 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: _finish_auth 0 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: _check_auth_tickets 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbfa8001670 con 0x7fbfac0b4d60 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb0cd5640 10 monclient: dump: 2024-03-25T18:20:24.573 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb3f3b640 5 monclient: authenticate success, global_id 4524 2024-03-25T18:20:24.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.571+0000 7fbfb3f3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfa80017d0 con 0x7fbfac0b4d60 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb0cd5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb0cd5640 1 -- 172.21.15.132:0/938996997 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbfa8001be0 con 0x7fbfac0b4d60 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 4 set_mon_vals no callback set 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb0cd5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb0cd5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb0cd5640 10 monclient: dump: 2024-03-25T18:20:24.580 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:24.581 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 10 monclient: shutdown 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/938996997 >> v1:172.21.15.132:6789/0 conn(0x7fbfac0b4d60 legacy=0x7fbfac13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/938996997 shutdown_connections 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.579+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/938996997 >> 172.21.15.132:0/938996997 conn(0x7fbfac0617a0 msgr2=0x7fbfac061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.580+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/938996997 shutdown_connections 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.580+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/938996997 wait complete. 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.580+0000 7fbfb3f3b640 10 monclient: build_initial_monmap 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.580+0000 7fbfb3f3b640 10 monclient: monmap: 2024-03-25T18:20:24.583 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 librados: starting msgr at 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 librados: starting objecter 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 Processor -- start 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 -- start start 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 librados: setting wanted keys 2024-03-25T18:20:24.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 librados: calling monclient init 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient: init 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient(hunting): picked mon.noname-a con 0x7fbfac0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient(hunting): picked mon.noname-c con 0x7fbfac0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient(hunting): picked mon.noname-b con 0x7fbfac14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac1d7ad0 con 0x7fbfac0b1730 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac1d77a0 con 0x7fbfac14be90 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfac1d8670 con 0x7fbfac0b4d60 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:24.585 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.581+0000 7fbfb3f3b640 10 monclient(hunting): authenticate will time out at 7664.844663s 2024-03-25T18:20:24.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.589+0000 7fbfb24d8640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbfac14be90 0x7fbfac1e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:60674/0 (socket says 172.21.15.132:60674) 2024-03-25T18:20:24.590 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.589+0000 7fbfb24d8640 1 -- 172.21.15.132:0/3734130411 learned_addr learned my addr 172.21.15.132:0/3734130411 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:24.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.590+0000 7fbfa27fc640 10 client.?.objecter ms_handle_connect 0x7fbfac14be90 2024-03-25T18:20:24.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.590+0000 7fbfa27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.598+0000 7fbfa27fc640 10 client.?.objecter ms_handle_connect 0x7fbfac0b1730 2024-03-25T18:20:24.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.598+0000 7fbfa27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.605+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3396170738 0 0) 0x7fbfac1d77a0 con 0x7fbfac14be90 2024-03-25T18:20:24.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.605+0000 7fbfa27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.605+0000 7fbfa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.605+0000 7fbfa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.605+0000 7fbfa27fc640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:20:24.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.605+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf90004eb0 con 0x7fbfac14be90 2024-03-25T18:20:24.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.613+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107261155 0 0) 0x7fbfac1d7ad0 con 0x7fbfac0b1730 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.613+0000 7fbfa27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.613+0000 7fbfa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.613+0000 7fbfa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.613+0000 7fbfa27fc640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.613+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfac1d77a0 con 0x7fbfac0b1730 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.614+0000 7fbfa27fc640 10 client.?.objecter ms_handle_connect 0x7fbfac0b4d60 2024-03-25T18:20:24.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.614+0000 7fbfa27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:24.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 844752642 0 0) 0x7fbf90004eb0 con 0x7fbfac14be90 2024-03-25T18:20:24.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbfac1d7ad0 con 0x7fbfac14be90 2024-03-25T18:20:24.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbfa4001c30 con 0x7fbfac14be90 2024-03-25T18:20:24.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:24.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 10 monclient(hunting): dump: 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.621+0000 7fbfa27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.628+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1650859402 0 0) 0x7fbfac1d77a0 con 0x7fbfac0b1730 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.628+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf90004eb0 con 0x7fbfac0b1730 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.628+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbfa8001b20 con 0x7fbfac0b1730 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.628+0000 7fbfa27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.628+0000 7fbfa27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.628+0000 7fbfa27fc640 10 monclient(hunting): dump: 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.629 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.630 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:24.630 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:24.630 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:24.630 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:24.630 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4005101233 0 0) 0x7fbfac1d7ad0 con 0x7fbfac14be90 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 >> v1:172.21.15.132:6790/0 conn(0x7fbfac0b4d60 legacy=0x7fbfac137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 >> v1:172.21.15.132:6789/0 conn(0x7fbfac0b1730 legacy=0x7fbfac1d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 1 monclient: found mon.b 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.633+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac0aaa20 con 0x7fbfac14be90 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfa27fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:24.635 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfa27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 10 monclient: _renew_subs 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfac143530 con 0x7fbfac14be90 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 10 client.4534.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 10 monclient: _renew_subs 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfac1432c0 con 0x7fbfac14be90 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.634+0000 7fbfb3f3b640 1 librados: init done 2024-03-25T18:20:24.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.635+0000 7fbfb3f3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.635+0000 7fbfb3f3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.635+0000 7fbfb3f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:24.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.635+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfac0627b0 con 0x7fbfac14be90 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.912+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbfa40020e0 con 0x7fbfac14be90 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fbfa4003b70 con 0x7fbfac14be90 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa27fc640 10 monclient: dump: 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.914 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa3fff640 4 set_mon_vals no callback set 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.913+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fbfa4003d10 con 0x7fbfac14be90 2024-03-25T18:20:24.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.914+0000 7fbfb1cd7640 10 monclient: get_auth_request con 0x7fbf9006ded0 auth_method 0 2024-03-25T18:20:24.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.914+0000 7fbfa27fc640 10 client.4534.objecter ms_handle_connect 0x7fbf9006ded0 2024-03-25T18:20:24.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.918+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fbfa4054a40 con 0x7fbfac14be90 2024-03-25T18:20:24.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.918+0000 7fbfa27fc640 10 client.4534.objecter ms_dispatch 0x7fbfac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:24.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.918+0000 7fbfa27fc640 3 client.4534.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:24.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.918+0000 7fbfa27fc640 3 client.4534.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:24.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.918+0000 7fbfa27fc640 20 client.4534.objecter dump_active .. 0 homeless 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.919+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbfa4050380 con 0x7fbfac14be90 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.919+0000 7fbfa27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:24.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:24.919+0000 7fbfa27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:25.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.175+0000 7fbfb3f3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.175+0000 7fbfb3f3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.175+0000 7fbfb3f3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:25.176 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.175+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfac069f40 con 0x7fbfac14be90 2024-03-25T18:20:25.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.193+0000 7fbfa27fc640 1 -- 172.21.15.132:0/3734130411 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fbfa4052480 con 0x7fbfac14be90 2024-03-25T18:20:25.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.193+0000 7fbfa27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:25.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.193+0000 7fbfa27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:25.194 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:25.195 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 10 librados: watch_flush enter 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 10 librados: watch_flush exit 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 20 client.4534.objecter shutdown clearing up homeless session... 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 10 client.4534.objecter successfully canceled tick 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 >> v1:172.21.15.161:6800/411950978 conn(0x7fbf9006ded0 legacy=0x7fbf90070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 10 monclient: shutdown 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.195+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 >> v1:172.21.15.161:6789/0 conn(0x7fbfac14be90 legacy=0x7fbfac1e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.196+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 shutdown_connections 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.196+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 >> 172.21.15.132:0/3734130411 conn(0x7fbfac0617a0 msgr2=0x7fbfac13c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:25.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.196+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 shutdown_connections 2024-03-25T18:20:25.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.196+0000 7fbfb3f3b640 1 -- 172.21.15.132:0/3734130411 wait complete. 2024-03-25T18:20:25.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:25.196+0000 7fbfb3f3b640 1 librados: shutdown 2024-03-25T18:20:25.209 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:32.015 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:32.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.130+0000 7f1ff90b5640 10 monclient: get_monmap_and_config 2024-03-25T18:20:32.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.130+0000 7f1ff90b5640 10 monclient: build_initial_monmap 2024-03-25T18:20:32.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.131+0000 7f1ff90b5640 10 monclient: monmap: 2024-03-25T18:20:32.132 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:32.132 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:32.132 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:32.132083+0000 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:32.132083+0000 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:32.133 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.131+0000 7f1ff90b5640 1 Processor -- start 2024-03-25T18:20:32.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.131+0000 7f1ff90b5640 1 -- start start 2024-03-25T18:20:32.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient: init 2024-03-25T18:20:32.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:32.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff40a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff413f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff41430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff4142520 con 0x7f1ff40a98f0 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40b28a0 con 0x7f1ff413f710 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40b2a30 con 0x7f1ff41430c0 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:32.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.132+0000 7f1ff90b5640 10 monclient(hunting): authenticate will time out at 7672.395513s 2024-03-25T18:20:32.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.142+0000 7f1ff259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1ff413f710 0x7f1ff41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:37260/0 (socket says 172.21.15.132:37260) 2024-03-25T18:20:32.144 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.142+0000 7f1ff259c640 1 -- 172.21.15.132:0/2597977779 learned_addr learned my addr 172.21.15.132:0/2597977779 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:32.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.155+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315838562 0 0) 0x7f1ff40b28a0 con 0x7f1ff413f710 2024-03-25T18:20:32.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.155+0000 7f1ff1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.155+0000 7f1ff1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.155+0000 7f1ff1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.155+0000 7f1ff1d9b640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:20:32.156 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.155+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc003520 con 0x7f1ff413f710 2024-03-25T18:20:32.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.160+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662923430 0 0) 0x7f1ff4142520 con 0x7f1ff40a98f0 2024-03-25T18:20:32.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.160+0000 7f1ff1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.160+0000 7f1ff1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.160+0000 7f1ff1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.160+0000 7f1ff1d9b640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:20:32.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.160+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff40b28a0 con 0x7f1ff40a98f0 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809552613 0 0) 0x7f1fdc003520 con 0x7f1ff413f710 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 >> v1:172.21.15.132:6790/0 conn(0x7f1ff41430c0 legacy=0x7f1ff414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 >> v1:172.21.15.132:6789/0 conn(0x7f1ff40a98f0 legacy=0x7f1ff40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:20:32.171 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40b44c0 con 0x7f1ff413f710 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1fe8001690 con 0x7f1ff413f710 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.169+0000 7f1ff1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.170+0000 7f1ff1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.170+0000 7f1ff1d9b640 10 monclient: dump: 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.172 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:32.173 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:32.173 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:32.173 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:32.173 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.170+0000 7f1ff90b5640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:20:32.173 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.170+0000 7f1ff90b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:32.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fe80017f0 con 0x7f1ff413f710 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff1d9b640 1 -- 172.21.15.132:0/2597977779 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1fe8001c00 con 0x7f1ff413f710 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff90b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff90b5640 4 set_mon_vals no callback set 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.185+0000 7f1ff1d9b640 10 monclient: dump: 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.187 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:32.188 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:32.188 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:32.188 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:32.188 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 10 monclient: shutdown 2024-03-25T18:20:32.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:32.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 1 -- 172.21.15.132:0/2597977779 >> v1:172.21.15.161:6789/0 conn(0x7f1ff413f710 legacy=0x7f1ff41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 1 -- 172.21.15.132:0/2597977779 shutdown_connections 2024-03-25T18:20:32.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 1 -- 172.21.15.132:0/2597977779 >> 172.21.15.132:0/2597977779 conn(0x7f1ff40b7ea0 msgr2=0x7f1ff40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:32.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 1 -- 172.21.15.132:0/2597977779 shutdown_connections 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.186+0000 7f1ff90b5640 1 -- 172.21.15.132:0/2597977779 wait complete. 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.187+0000 7f1ff90b5640 10 monclient: build_initial_monmap 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.187+0000 7f1ff90b5640 10 monclient: monmap: 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:32.191 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.187+0000 7f1ff90b5640 1 librados: starting msgr at 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.187+0000 7f1ff90b5640 1 librados: starting objecter 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 Processor -- start 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 -- start start 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 librados: setting wanted keys 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 librados: calling monclient init 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient: init 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient(hunting): picked mon.noname-a con 0x7f1ff40a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:32.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient(hunting): picked mon.noname-c con 0x7f1ff413f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:32.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient(hunting): picked mon.noname-b con 0x7f1ff41430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:32.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff4138a50 con 0x7f1ff40a98f0 2024-03-25T18:20:32.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40ab9e0 con 0x7f1ff41430c0 2024-03-25T18:20:32.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff41d7880 con 0x7f1ff413f710 2024-03-25T18:20:32.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:32.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.188+0000 7f1ff90b5640 10 monclient(hunting): authenticate will time out at 7672.451701s 2024-03-25T18:20:32.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.191+0000 7f1ff2d9d640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f1ff40a98f0 0x7f1ff4138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:36888/0 (socket says 172.21.15.132:36888) 2024-03-25T18:20:32.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.191+0000 7f1ff2d9d640 1 -- 172.21.15.132:0/1004151705 learned_addr learned my addr 172.21.15.132:0/1004151705 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:32.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.192+0000 7f1fdb7fe640 10 client.?.objecter ms_handle_connect 0x7f1ff41430c0 2024-03-25T18:20:32.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.192+0000 7f1fdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.193+0000 7f1fdb7fe640 10 client.?.objecter ms_handle_connect 0x7f1ff40a98f0 2024-03-25T18:20:32.194 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.193+0000 7f1fdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.251+0000 7f1fdb7fe640 10 client.?.objecter ms_handle_connect 0x7f1ff413f710 2024-03-25T18:20:32.252 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.251+0000 7f1fdb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:32.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.271+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817838304 0 0) 0x7f1ff41d7880 con 0x7f1ff413f710 2024-03-25T18:20:32.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.271+0000 7f1fdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.271+0000 7f1fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.271+0000 7f1fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.271+0000 7f1fdb7fe640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:20:32.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.271+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0004eb0 con 0x7f1ff413f710 2024-03-25T18:20:32.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.273+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525298397 0 0) 0x7f1ff40ab9e0 con 0x7f1ff41430c0 2024-03-25T18:20:32.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.273+0000 7f1fdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.273+0000 7f1fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.273+0000 7f1fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.273+0000 7f1fdb7fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:20:32.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.273+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff41d7880 con 0x7f1ff41430c0 2024-03-25T18:20:32.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4194002289 0 0) 0x7f1fe0004eb0 con 0x7f1ff413f710 2024-03-25T18:20:32.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ff40ab9e0 con 0x7f1ff413f710 2024-03-25T18:20:32.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1fe8001b00 con 0x7f1ff413f710 2024-03-25T18:20:32.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:32.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 10 monclient(hunting): dump: 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.292 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.290+0000 7f1fdb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.303+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611008087 0 0) 0x7f1ff41d7880 con 0x7f1ff41430c0 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.303+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fe0004eb0 con 0x7f1ff41430c0 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.303+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1fec001b10 con 0x7f1ff41430c0 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.303+0000 7f1fdb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.303+0000 7f1fdb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.303+0000 7f1fdb7fe640 10 monclient(hunting): dump: 2024-03-25T18:20:32.304 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:32.305 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.317+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1446207106 0 0) 0x7f1ff4138a50 con 0x7f1ff40a98f0 2024-03-25T18:20:32.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.317+0000 7f1fdb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:32.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.317+0000 7f1fdb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:32.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.317+0000 7f1fdb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:32.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.317+0000 7f1fdb7fe640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:20:32.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.317+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff41d7880 con 0x7f1ff40a98f0 2024-03-25T18:20:32.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3878460644 0 0) 0x7f1ff40ab9e0 con 0x7f1ff413f710 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 >> v1:172.21.15.161:6789/0 conn(0x7f1ff41430c0 legacy=0x7f1ff41e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 >> v1:172.21.15.132:6789/0 conn(0x7f1ff40a98f0 legacy=0x7f1ff4138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 1 monclient: found mon.c 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff41d84b0 con 0x7f1ff413f710 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1fdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 5 monclient: authenticate success, global_id 4532 2024-03-25T18:20:32.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 10 monclient: _renew_subs 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff41d51f0 con 0x7f1ff413f710 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 10 client.4532.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 10 monclient: _renew_subs 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff4138e60 con 0x7f1ff413f710 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.319+0000 7f1ff90b5640 1 librados: init done 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.320+0000 7f1ff90b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.320+0000 7f1ff90b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.320+0000 7f1ff90b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:32.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.320+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff41d76e0 con 0x7f1ff413f710 2024-03-25T18:20:32.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1fe8001220 con 0x7f1ff413f710 2024-03-25T18:20:32.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1fdb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:32.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1fe8004a90 con 0x7f1ff413f710 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1fdb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1fdb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1fdb7fe640 10 monclient: dump: 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:32.642 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:32.643 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:32.643 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:32.643 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.640+0000 7f1ff159a640 4 set_mon_vals no callback set 2024-03-25T18:20:32.643 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.641+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f1fe801aa60 con 0x7f1ff413f710 2024-03-25T18:20:32.643 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.641+0000 7f1ff2d9d640 10 monclient: get_auth_request con 0x7f1fe006df50 auth_method 0 2024-03-25T18:20:32.643 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.642+0000 7f1fdb7fe640 10 client.4532.objecter ms_handle_connect 0x7f1fe006df50 2024-03-25T18:20:32.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f1fe80546e0 con 0x7f1ff413f710 2024-03-25T18:20:32.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 10 client.4532.objecter ms_dispatch 0x7f1ff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:32.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 3 client.4532.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:32.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 3 client.4532.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:32.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 20 client.4532.objecter dump_active .. 0 homeless 2024-03-25T18:20:32.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1fe8050020 con 0x7f1ff413f710 2024-03-25T18:20:32.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:32.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.650+0000 7f1fdb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:32.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.902+0000 7f1ff90b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:32.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.902+0000 7f1ff90b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:32.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.902+0000 7f1ff90b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:32.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.902+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ff40627b0 con 0x7f1ff413f710 2024-03-25T18:20:32.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.968+0000 7f1fdb7fe640 1 -- 172.21.15.132:0/1004151705 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f1fe8052120 con 0x7f1ff413f710 2024-03-25T18:20:32.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.968+0000 7f1fdb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:32.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.968+0000 7f1fdb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:32.969 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:32.970 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 10 librados: watch_flush enter 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 10 librados: watch_flush exit 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 20 client.4532.objecter shutdown clearing up homeless session... 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 10 client.4532.objecter successfully canceled tick 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 >> v1:172.21.15.161:6800/411950978 conn(0x7f1fe006df50 legacy=0x7f1fe0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 10 monclient: shutdown 2024-03-25T18:20:32.972 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:32.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.970+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 >> v1:172.21.15.132:6790/0 conn(0x7f1ff413f710 legacy=0x7f1ff41d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:32.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.971+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 shutdown_connections 2024-03-25T18:20:32.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.971+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 >> 172.21.15.132:0/1004151705 conn(0x7f1ff40b7ea0 msgr2=0x7f1ff4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:32.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.971+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 shutdown_connections 2024-03-25T18:20:32.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.971+0000 7f1ff90b5640 1 -- 172.21.15.132:0/1004151705 wait complete. 2024-03-25T18:20:32.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:32.971+0000 7f1ff90b5640 1 librados: shutdown 2024-03-25T18:20:32.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:35.158 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:20.422 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.217 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:20.481 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.260 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:20.502 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.264 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:20.528 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.269 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:20.511 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.276 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:20.541 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.285 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:20.549 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.293 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:20.534 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:35.307 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:20.571 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:39.790 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.902+0000 7fc6cdc50640 10 monclient: get_monmap_and_config 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.902+0000 7fc6cdc50640 10 monclient: build_initial_monmap 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.902+0000 7fc6cdc50640 10 monclient: monmap: 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:39.903531+0000 2024-03-25T18:20:39.903 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:39.903531+0000 2024-03-25T18:20:39.904 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:39.904 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:39.904 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:39.904 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:39.904 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:39.904 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.903+0000 7fc6cdc50640 1 Processor -- start 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.903+0000 7fc6cdc50640 1 -- start start 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.903+0000 7fc6cdc50640 10 monclient: init 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient(hunting): picked mon.noname-b con 0x7fc6c80b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient(hunting): picked mon.noname-c con 0x7fc6c80b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:39.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient(hunting): picked mon.noname-a con 0x7fc6c814be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:39.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c80af940 con 0x7fc6c814be90 2024-03-25T18:20:39.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c80b0a70 con 0x7fc6c80b1730 2024-03-25T18:20:39.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c80b0d50 con 0x7fc6c80b4d60 2024-03-25T18:20:39.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:39.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.904+0000 7fc6cdc50640 10 monclient(hunting): authenticate will time out at 7680.166906s 2024-03-25T18:20:39.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.915+0000 7fc6c6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc6c80b1730 0x7fc6c80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:36378/0 (socket says 172.21.15.132:36378) 2024-03-25T18:20:39.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.915+0000 7fc6c6ffd640 1 -- 172.21.15.132:0/898254531 learned_addr learned my addr 172.21.15.132:0/898254531 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:39.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.929+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364799194 0 0) 0x7fc6c80b0a70 con 0x7fc6c80b1730 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.929+0000 7fc6c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.929+0000 7fc6c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.929+0000 7fc6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.929+0000 7fc6c67fc640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.929+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b0003520 con 0x7fc6c80b1730 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647519232 0 0) 0x7fc6c80af940 con 0x7fc6c814be90 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:20:39.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c80b0a70 con 0x7fc6c814be90 2024-03-25T18:20:39.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533766333 0 0) 0x7fc6c80b0d50 con 0x7fc6c80b4d60 2024-03-25T18:20:39.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:39.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:39.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:39.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:20:39.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.930+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c80af940 con 0x7fc6c80b4d60 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1616595543 0 0) 0x7fc6c80b0a70 con 0x7fc6c814be90 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 >> v1:172.21.15.132:6790/0 conn(0x7fc6c80b4d60 legacy=0x7fc6c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 >> v1:172.21.15.161:6789/0 conn(0x7fc6c80b1730 legacy=0x7fc6c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 1 monclient: found mon.noname-a 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c80b1030 con 0x7fc6c814be90 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:39.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc6c00019f0 con 0x7fc6c814be90 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6cdc50640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6cdc50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6c67fc640 10 monclient: dump: 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:39.944 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:39.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.941+0000 7fc6cdc50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:39.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6c0001b50 con 0x7fc6c814be90 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6c67fc640 1 -- 172.21.15.132:0/898254531 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc6c0001f60 con 0x7fc6c814be90 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6cdc50640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6cdc50640 4 set_mon_vals no callback set 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.958+0000 7fc6c67fc640 10 monclient: dump: 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:39.960 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:39.961 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:39.961 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 10 monclient: shutdown 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 1 -- 172.21.15.132:0/898254531 >> v1:172.21.15.132:6789/0 conn(0x7fc6c814be90 legacy=0x7fc6c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 1 -- 172.21.15.132:0/898254531 shutdown_connections 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 1 -- 172.21.15.132:0/898254531 >> 172.21.15.132:0/898254531 conn(0x7fc6c80617a0 msgr2=0x7fc6c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 1 -- 172.21.15.132:0/898254531 shutdown_connections 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.959+0000 7fc6cdc50640 1 -- 172.21.15.132:0/898254531 wait complete. 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.960+0000 7fc6cdc50640 10 monclient: build_initial_monmap 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.960+0000 7fc6cdc50640 10 monclient: monmap: 2024-03-25T18:20:39.962 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.960+0000 7fc6cdc50640 1 librados: starting msgr at 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.960+0000 7fc6cdc50640 1 librados: starting objecter 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 Processor -- start 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 -- start start 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 librados: setting wanted keys 2024-03-25T18:20:39.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 librados: calling monclient init 2024-03-25T18:20:39.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient: init 2024-03-25T18:20:39.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient(hunting): picked mon.noname-a con 0x7fc6c80b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient(hunting): picked mon.noname-b con 0x7fc6c80b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient(hunting): picked mon.noname-c con 0x7fc6c814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c81d7730 con 0x7fc6c80b1730 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c81d85b0 con 0x7fc6c80b4d60 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6c81d7980 con 0x7fc6c814be90 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:39.965 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.961+0000 7fc6cdc50640 10 monclient(hunting): authenticate will time out at 7680.224605s 2024-03-25T18:20:39.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.980+0000 7fc6c6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc6c80b4d60 0x7fc6c813a0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:36426/0 (socket says 172.21.15.132:36426) 2024-03-25T18:20:39.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.980+0000 7fc6c6ffd640 1 -- 172.21.15.132:0/1665956674 learned_addr learned my addr 172.21.15.132:0/1665956674 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:39.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.981+0000 7fc6affff640 10 client.?.objecter ms_handle_connect 0x7fc6c80b1730 2024-03-25T18:20:39.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.981+0000 7fc6affff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:39.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.981+0000 7fc6affff640 10 client.?.objecter ms_handle_connect 0x7fc6c814be90 2024-03-25T18:20:39.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.981+0000 7fc6affff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:39.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.983+0000 7fc6affff640 10 client.?.objecter ms_handle_connect 0x7fc6c80b4d60 2024-03-25T18:20:39.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.983+0000 7fc6affff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:39.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.989+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686105138 0 0) 0x7fc6c81d7730 con 0x7fc6c80b1730 2024-03-25T18:20:39.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.989+0000 7fc6affff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:39.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.989+0000 7fc6affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:39.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.989+0000 7fc6affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:39.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.989+0000 7fc6affff640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:20:39.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:39.989+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b4004eb0 con 0x7fc6c80b1730 2024-03-25T18:20:40.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.008+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020931344 0 0) 0x7fc6c81d7980 con 0x7fc6c814be90 2024-03-25T18:20:40.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.008+0000 7fc6affff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.008+0000 7fc6affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.008+0000 7fc6affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.008+0000 7fc6affff640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:20:40.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.008+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c81d7730 con 0x7fc6c814be90 2024-03-25T18:20:40.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.016+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724656383 0 0) 0x7fc6c81d85b0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.016+0000 7fc6affff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.016+0000 7fc6affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.016+0000 7fc6affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.016+0000 7fc6affff640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:20:40.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.016+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c81d7980 con 0x7fc6c80b4d60 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2502114773 0 0) 0x7fc6c81d7980 con 0x7fc6c80b4d60 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c81d85b0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc6bc001950 con 0x7fc6c80b4d60 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 10 monclient(hunting): dump: 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:40.023 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:40.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.022+0000 7fc6affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.029+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408045184 0 0) 0x7fc6b4004eb0 con 0x7fc6c80b1730 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.029+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c81d7980 con 0x7fc6c80b1730 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.029+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc6b80018f0 con 0x7fc6c80b1730 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.029+0000 7fc6affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.029+0000 7fc6affff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.029+0000 7fc6affff640 10 monclient(hunting): dump: 2024-03-25T18:20:40.030 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:40.031 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:40.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 248274877 0 0) 0x7fc6c81d85b0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 >> v1:172.21.15.132:6790/0 conn(0x7fc6c814be90 legacy=0x7fc6c81e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 >> v1:172.21.15.132:6789/0 conn(0x7fc6c80b1730 legacy=0x7fc6c81d7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 1 monclient: found mon.b 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c81394d0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:20:40.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 10 monclient: _renew_subs 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c813a7d0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 10 monclient: _renew_subs 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c813d2d0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.036+0000 7fc6cdc50640 1 librados: init done 2024-03-25T18:20:40.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.037+0000 7fc6cdc50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.037+0000 7fc6cdc50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.037+0000 7fc6cdc50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:40.040 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.037+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c813ab20 con 0x7fc6c80b4d60 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6bc0047e0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc6bc001d80 con 0x7fc6c80b4d60 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6affff640 10 monclient: dump: 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:40.341 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:40.342 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.340+0000 7fc6c5ffb640 4 set_mon_vals no callback set 2024-03-25T18:20:40.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.341+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fc6bc01a9e0 con 0x7fc6c80b4d60 2024-03-25T18:20:40.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.341+0000 7fc6c77fe640 10 monclient: get_auth_request con 0x7fc6b406df70 auth_method 0 2024-03-25T18:20:40.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.341+0000 7fc6affff640 10 client.4546.objecter ms_handle_connect 0x7fc6b406df70 2024-03-25T18:20:40.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fc6bc054810 con 0x7fc6c80b4d60 2024-03-25T18:20:40.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 10 client.4546.objecter ms_dispatch 0x7fc6c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:40.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:40.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:40.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:20:40.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc6bc050150 con 0x7fc6c80b4d60 2024-03-25T18:20:40.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.345+0000 7fc6affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:40.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.568+0000 7fc6cdc50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.568+0000 7fc6cdc50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.568+0000 7fc6cdc50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:40.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.568+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6c81d7b10 con 0x7fc6c80b4d60 2024-03-25T18:20:40.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.605+0000 7fc6affff640 1 -- 172.21.15.132:0/1665956674 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fc6bc052250 con 0x7fc6c80b4d60 2024-03-25T18:20:40.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.605+0000 7fc6affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:40.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.605+0000 7fc6affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:40.606 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:40.606 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:40.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.607+0000 7fc6cdc50640 10 librados: watch_flush enter 2024-03-25T18:20:40.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.607+0000 7fc6cdc50640 10 librados: watch_flush exit 2024-03-25T18:20:40.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.607+0000 7fc6cdc50640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:20:40.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.607+0000 7fc6cdc50640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:20:40.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 >> v1:172.21.15.161:6800/411950978 conn(0x7fc6b406df70 legacy=0x7fc6b4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 10 monclient: shutdown 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 >> v1:172.21.15.161:6789/0 conn(0x7fc6c80b4d60 legacy=0x7fc6c813a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 shutdown_connections 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 >> 172.21.15.132:0/1665956674 conn(0x7fc6c80617a0 msgr2=0x7fc6c813c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 shutdown_connections 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 -- 172.21.15.132:0/1665956674 wait complete. 2024-03-25T18:20:40.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:40.608+0000 7fc6cdc50640 1 librados: shutdown 2024-03-25T18:20:40.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:47.428 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.541+0000 7f39964ba640 10 monclient: get_monmap_and_config 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.541+0000 7f39964ba640 10 monclient: build_initial_monmap 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.541+0000 7f39964ba640 10 monclient: monmap: 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:47.542314+0000 2024-03-25T18:20:47.542 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:47.542314+0000 2024-03-25T18:20:47.543 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:47.543 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:47.543 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:47.543 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:47.543 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:47.543 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.542+0000 7f39964ba640 1 Processor -- start 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.542+0000 7f39964ba640 1 -- start start 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.542+0000 7f39964ba640 10 monclient: init 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.542+0000 7f39964ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.542+0000 7f39964ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.542+0000 7f39964ba640 10 monclient(hunting): picked mon.noname-b con 0x7f39900b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 10 monclient(hunting): picked mon.noname-c con 0x7f39900b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 10 monclient(hunting): picked mon.noname-a con 0x7f399014be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900af940 con 0x7f399014be90 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900b0a70 con 0x7f39900b1730 2024-03-25T18:20:47.544 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39900b0d50 con 0x7f39900b4d60 2024-03-25T18:20:47.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:47.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.543+0000 7f39964ba640 10 monclient(hunting): authenticate will time out at 7687.805779s 2024-03-25T18:20:47.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.554+0000 7f398ffff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f39900b4d60 0x7f399013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:55862/0 (socket says 172.21.15.132:55862) 2024-03-25T18:20:47.555 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.554+0000 7f398ffff640 1 -- 172.21.15.132:0/1369382323 learned_addr learned my addr 172.21.15.132:0/1369382323 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:47.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.570+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269249768 0 0) 0x7f39900af940 con 0x7f399014be90 2024-03-25T18:20:47.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.570+0000 7f398effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.570+0000 7f398effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.570+0000 7f398effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.570+0000 7f398effd640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:20:47.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.570+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3978003520 con 0x7f399014be90 2024-03-25T18:20:47.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590796427 0 0) 0x7f39900b0d50 con 0x7f39900b4d60 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900af940 con 0x7f39900b4d60 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587901222 0 0) 0x7f39900b0a70 con 0x7f39900b1730 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:20:47.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.578+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900b0d50 con 0x7f39900b1730 2024-03-25T18:20:47.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.585+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3380247518 0 0) 0x7f3978003520 con 0x7f399014be90 2024-03-25T18:20:47.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.585+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 >> v1:172.21.15.132:6790/0 conn(0x7f39900b4d60 legacy=0x7f399013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.585+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 >> v1:172.21.15.161:6789/0 conn(0x7f39900b1730 legacy=0x7f39900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.585+0000 7f398effd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:47.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.585+0000 7f398effd640 1 monclient: found mon.noname-a 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39900b1030 con 0x7f399014be90 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 10 monclient: _finish_auth 0 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3988001a10 con 0x7f399014be90 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f398effd640 10 monclient: dump: 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.588 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f39964ba640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:20:47.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.586+0000 7f39964ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:47.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3988001b70 con 0x7f399014be90 2024-03-25T18:20:47.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f398effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:47.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f398effd640 1 -- 172.21.15.132:0/1369382323 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3988001f80 con 0x7f399014be90 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f398effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f398effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f398effd640 10 monclient: dump: 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:47.592 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f39964ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:47.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.590+0000 7f39964ba640 4 set_mon_vals no callback set 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 10 monclient: shutdown 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 1 -- 172.21.15.132:0/1369382323 >> v1:172.21.15.132:6789/0 conn(0x7f399014be90 legacy=0x7f399014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 1 -- 172.21.15.132:0/1369382323 shutdown_connections 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 1 -- 172.21.15.132:0/1369382323 >> 172.21.15.132:0/1369382323 conn(0x7f39900617a0 msgr2=0x7f3990061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 1 -- 172.21.15.132:0/1369382323 shutdown_connections 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.591+0000 7f39964ba640 1 -- 172.21.15.132:0/1369382323 wait complete. 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 10 monclient: build_initial_monmap 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 10 monclient: monmap: 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:47.595 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 1 librados: starting msgr at 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 1 librados: starting objecter 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 1 Processor -- start 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 1 -- start start 2024-03-25T18:20:47.596 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 1 librados: setting wanted keys 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 1 librados: calling monclient init 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.592+0000 7f39964ba640 10 monclient: init 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient(hunting): picked mon.noname-a con 0x7f39900b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient(hunting): picked mon.noname-c con 0x7f39900b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient(hunting): picked mon.noname-b con 0x7f399014be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3990141950 con 0x7f39900b1730 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39901d7940 con 0x7f399014be90 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39901d8570 con 0x7f39900b4d60 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:47.597 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.593+0000 7f39964ba640 10 monclient(hunting): authenticate will time out at 7687.856301s 2024-03-25T18:20:47.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.604+0000 7f3994a57640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f399014be90 0x7f39901e2fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:36628/0 (socket says 172.21.15.132:36628) 2024-03-25T18:20:47.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.604+0000 7f3994a57640 1 -- 172.21.15.132:0/1289717375 learned_addr learned my addr 172.21.15.132:0/1289717375 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:47.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.605+0000 7f398cff9640 10 client.?.objecter ms_handle_connect 0x7f399014be90 2024-03-25T18:20:47.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.605+0000 7f398cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:47.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.609+0000 7f398cff9640 10 client.?.objecter ms_handle_connect 0x7f39900b1730 2024-03-25T18:20:47.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.609+0000 7f398cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:47.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.609+0000 7f398cff9640 10 client.?.objecter ms_handle_connect 0x7f39900b4d60 2024-03-25T18:20:47.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.609+0000 7f398cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:47.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.613+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114364847 0 0) 0x7f39901d8570 con 0x7f39900b4d60 2024-03-25T18:20:47.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.613+0000 7f398cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.613+0000 7f398cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.613+0000 7f398cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.613+0000 7f398cff9640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:20:47.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.613+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c004eb0 con 0x7f39900b4d60 2024-03-25T18:20:47.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.623+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621479309 0 0) 0x7f39901d7940 con 0x7f399014be90 2024-03-25T18:20:47.624 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.623+0000 7f398cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.623+0000 7f398cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.623+0000 7f398cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.623+0000 7f398cff9640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:20:47.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.623+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39901d8570 con 0x7f399014be90 2024-03-25T18:20:47.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.627+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392055335 0 0) 0x7f3990141950 con 0x7f39900b1730 2024-03-25T18:20:47.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.627+0000 7f398cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:47.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.627+0000 7f398cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:47.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.627+0000 7f398cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:47.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.627+0000 7f398cff9640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:20:47.629 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.627+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39901d7940 con 0x7f39900b1730 2024-03-25T18:20:47.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444728022 0 0) 0x7f397c004eb0 con 0x7f39900b4d60 2024-03-25T18:20:47.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3990141950 con 0x7f39900b4d60 2024-03-25T18:20:47.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f39840018f0 con 0x7f39900b4d60 2024-03-25T18:20:47.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:47.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 10 monclient(hunting): dump: 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.636+0000 7f398cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:47.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.641+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 696551216 0 0) 0x7f39901d8570 con 0x7f399014be90 2024-03-25T18:20:47.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.641+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f397c004eb0 con 0x7f399014be90 2024-03-25T18:20:47.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.641+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3988001e80 con 0x7f399014be90 2024-03-25T18:20:47.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.641+0000 7f398cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.641+0000 7f398cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.641+0000 7f398cff9640 10 monclient(hunting): dump: 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:47.643 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1343343616 0 0) 0x7f397c004eb0 con 0x7f399014be90 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 >> v1:172.21.15.132:6790/0 conn(0x7f39900b4d60 legacy=0x7f3990142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 >> v1:172.21.15.132:6789/0 conn(0x7f39900b1730 legacy=0x7f39901d7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 1 monclient: found mon.b 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3990138840 con 0x7f399014be90 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:47.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f398cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 10 monclient: _renew_subs 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3990142f70 con 0x7f399014be90 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 10 monclient: _renew_subs 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3990137da0 con 0x7f399014be90 2024-03-25T18:20:47.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.654+0000 7f39964ba640 1 librados: init done 2024-03-25T18:20:47.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.655+0000 7f39964ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:47.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.655+0000 7f39964ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:47.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.655+0000 7f39964ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:47.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.655+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39901d7ad0 con 0x7f399014be90 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.924+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3988001590 con 0x7f399014be90 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3988002100 con 0x7f399014be90 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398cff9640 10 monclient: dump: 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.926 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398e7fc640 4 set_mon_vals no callback set 2024-03-25T18:20:47.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.925+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f3988003e40 con 0x7f399014be90 2024-03-25T18:20:47.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.926+0000 7f398ffff640 10 monclient: get_auth_request con 0x7f397c06dfb0 auth_method 0 2024-03-25T18:20:47.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.926+0000 7f398cff9640 10 client.4552.objecter ms_handle_connect 0x7f397c06dfb0 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f3988054ab0 con 0x7f399014be90 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 10 client.4552.objecter ms_dispatch 0x7f3990000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39880505c0 con 0x7f399014be90 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:47.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:47.930+0000 7f398cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:48.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.168+0000 7f39964ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.168+0000 7f39964ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.168+0000 7f39964ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:20:48.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.168+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39901389d0 con 0x7f399014be90 2024-03-25T18:20:48.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.197+0000 7f398cff9640 1 -- 172.21.15.132:0/1289717375 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f39880524f0 con 0x7f399014be90 2024-03-25T18:20:48.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.197+0000 7f398cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.197+0000 7f398cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:48.198 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:48.199 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:48.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.199+0000 7f39964ba640 10 librados: watch_flush enter 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 10 librados: watch_flush exit 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 >> v1:172.21.15.161:6800/411950978 conn(0x7f397c06dfb0 legacy=0x7f397c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 10 monclient: shutdown 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 >> v1:172.21.15.161:6789/0 conn(0x7f399014be90 legacy=0x7f39901e2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 shutdown_connections 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 >> 172.21.15.132:0/1289717375 conn(0x7f39900617a0 msgr2=0x7f39900b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 shutdown_connections 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 -- 172.21.15.132:0/1289717375 wait complete. 2024-03-25T18:20:48.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:48.200+0000 7f39964ba640 1 librados: shutdown 2024-03-25T18:20:48.213 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:55.018 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:55.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.131+0000 7fb4f02a8640 10 monclient: get_monmap_and_config 2024-03-25T18:20:55.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.131+0000 7fb4f02a8640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.131+0000 7fb4f02a8640 10 monclient: monmap: 2024-03-25T18:20:55.132 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:20:55.132543+0000 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:20:55.132543+0000 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:20:55.133 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:55.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.132+0000 7fb4f02a8640 1 Processor -- start 2024-03-25T18:20:55.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.132+0000 7fb4f02a8640 1 -- start start 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.132+0000 7fb4f02a8640 10 monclient: init 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e80a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e81430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e8142520 con 0x7fb4e813f710 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80b28a0 con 0x7fb4e81430c0 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80b2a30 con 0x7fb4e80a98f0 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.133+0000 7fb4f02a8640 10 monclient(hunting): authenticate will time out at 7695.395938s 2024-03-25T18:20:55.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.136+0000 7fb4ee845640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb4e81430c0 0x7fb4e814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:47236/0 (socket says 172.21.15.132:47236) 2024-03-25T18:20:55.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.136+0000 7fb4ee845640 1 -- 172.21.15.132:0/3763301395 learned_addr learned my addr 172.21.15.132:0/3763301395 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:55.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.153+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695388601 0 0) 0x7fb4e80b28a0 con 0x7fb4e81430c0 2024-03-25T18:20:55.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.153+0000 7fb4ed042640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.153+0000 7fb4ed042640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.153+0000 7fb4ed042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.153+0000 7fb4ed042640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:20:55.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.153+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d4003520 con 0x7fb4e81430c0 2024-03-25T18:20:55.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.173+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852946230 0 0) 0x7fb4e80b2a30 con 0x7fb4e80a98f0 2024-03-25T18:20:55.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.173+0000 7fb4ed042640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.173+0000 7fb4ed042640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.173+0000 7fb4ed042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.173+0000 7fb4ed042640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:20:55.174 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.173+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e80b28a0 con 0x7fb4e80a98f0 2024-03-25T18:20:55.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.254+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109001603 0 0) 0x7fb4e8142520 con 0x7fb4e813f710 2024-03-25T18:20:55.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.254+0000 7fb4ed042640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.254+0000 7fb4ed042640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.254+0000 7fb4ed042640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.254+0000 7fb4ed042640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:20:55.255 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.254+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e80b2a30 con 0x7fb4e813f710 2024-03-25T18:20:55.262 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4045619896 0 0) 0x7fb4e80b28a0 con 0x7fb4e80a98f0 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 >> v1:172.21.15.161:6789/0 conn(0x7fb4e81430c0 legacy=0x7fb4e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 >> v1:172.21.15.132:6789/0 conn(0x7fb4e813f710 legacy=0x7fb4e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 1 monclient: found mon.noname-b 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e80b44c0 con 0x7fb4e80a98f0 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb4e4001670 con 0x7fb4e80a98f0 2024-03-25T18:20:55.263 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4f02a8640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4f02a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4ed042640 10 monclient: dump: 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:55.264 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:55.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.262+0000 7fb4f02a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4e40017d0 con 0x7fb4e80a98f0 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4ed042640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4ed042640 1 -- 172.21.15.132:0/3763301395 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb4e4001be0 con 0x7fb4e80a98f0 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4ed042640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4ed042640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4ed042640 10 monclient: dump: 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:55.267 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4f02a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:55.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.267+0000 7fb4f02a8640 4 set_mon_vals no callback set 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 10 monclient: shutdown 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 1 -- 172.21.15.132:0/3763301395 >> v1:172.21.15.132:6790/0 conn(0x7fb4e80a98f0 legacy=0x7fb4e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 1 -- 172.21.15.132:0/3763301395 shutdown_connections 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 1 -- 172.21.15.132:0/3763301395 >> 172.21.15.132:0/3763301395 conn(0x7fb4e80b7ea0 msgr2=0x7fb4e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 1 -- 172.21.15.132:0/3763301395 shutdown_connections 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 1 -- 172.21.15.132:0/3763301395 wait complete. 2024-03-25T18:20:55.270 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.268+0000 7fb4f02a8640 10 monclient: monmap: 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 1 librados: starting msgr at 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 1 librados: starting objecter 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 1 Processor -- start 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 1 -- start start 2024-03-25T18:20:55.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 1 librados: setting wanted keys 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 1 librados: calling monclient init 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.269+0000 7fb4f02a8640 10 monclient: init 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb4e80a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb4e813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb4e81430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80abfd0 con 0x7fb4e80a98f0 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80ac2b0 con 0x7fb4e813f710 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e80ac590 con 0x7fb4e81430c0 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.270+0000 7fb4f02a8640 10 monclient(hunting): authenticate will time out at 7695.531984s 2024-03-25T18:20:55.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.273+0000 7fb4ee044640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fb4e80a98f0 0x7fb4e80ac9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:56784/0 (socket says 172.21.15.132:56784) 2024-03-25T18:20:55.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.273+0000 7fb4ee044640 1 -- 172.21.15.132:0/73319301 learned_addr learned my addr 172.21.15.132:0/73319301 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:20:55.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.275+0000 7fb4deffd640 10 client.?.objecter ms_handle_connect 0x7fb4e80a98f0 2024-03-25T18:20:55.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.275+0000 7fb4deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.280 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.281+0000 7fb4deffd640 10 client.?.objecter ms_handle_connect 0x7fb4e81430c0 2024-03-25T18:20:55.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.281+0000 7fb4deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.288+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263494685 0 0) 0x7fb4e80ac590 con 0x7fb4e81430c0 2024-03-25T18:20:55.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.288+0000 7fb4deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.289 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.288+0000 7fb4deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.289 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.288+0000 7fb4deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.289 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.288+0000 7fb4deffd640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:20:55.289 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.288+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4cc004eb0 con 0x7fb4e81430c0 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1094165058 0 0) 0x7fb4cc004eb0 con 0x7fb4e81430c0 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e80ac590 con 0x7fb4e81430c0 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb4e0001af0 con 0x7fb4e81430c0 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 10 monclient(hunting): dump: 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:55.307 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:55.308 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:55.308 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:55.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.306+0000 7fb4deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:55.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.311+0000 7fb4deffd640 10 client.?.objecter ms_handle_connect 0x7fb4e813f710 2024-03-25T18:20:55.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.311+0000 7fb4deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1983280445 0 0) 0x7fb4e80ac590 con 0x7fb4e81430c0 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 >> v1:172.21.15.161:6789/0 conn(0x7fb4e813f710 legacy=0x7fb4e8138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 >> v1:172.21.15.132:6789/0 conn(0x7fb4e80a98f0 legacy=0x7fb4e80ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 1 monclient: found mon.c 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e81382a0 con 0x7fb4e81430c0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 10 monclient: _renew_subs 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e8139530 con 0x7fb4e81430c0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 10 monclient: _renew_subs 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e81e3480 con 0x7fb4e81430c0 2024-03-25T18:20:55.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.322+0000 7fb4f02a8640 1 librados: init done 2024-03-25T18:20:55.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.324+0000 7fb4f02a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.324+0000 7fb4f02a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.324+0000 7fb4f02a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:55.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.324+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e80627b0 con 0x7fb4e81430c0 2024-03-25T18:20:55.602 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:40.866 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4e0001fa0 con 0x7fb4e81430c0 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fb4e0004d00 con 0x7fb4e81430c0 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4deffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4deffd640 10 monclient: dump: 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.614 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.613+0000 7fb4ec841640 4 set_mon_vals no callback set 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.614+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fb4e0004f20 con 0x7fb4e81430c0 2024-03-25T18:20:55.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.615+0000 7fb4ee044640 10 monclient: get_auth_request con 0x7fb4cc06dc70 auth_method 0 2024-03-25T18:20:55.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.615+0000 7fb4deffd640 10 client.4550.objecter ms_handle_connect 0x7fb4cc06dc70 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.620+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fb4e00549f0 con 0x7fb4e81430c0 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.620+0000 7fb4deffd640 10 client.4550.objecter ms_dispatch 0x7fb4e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.620+0000 7fb4deffd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.620+0000 7fb4deffd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.621+0000 7fb4deffd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.621+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4e0050500 con 0x7fb4e81430c0 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.621+0000 7fb4deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:55.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.621+0000 7fb4deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:55.623 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:40.887 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.661 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:40.903 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.667 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:40.908 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.672 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:07:40.936 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.680 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:40.944 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.689 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:40.953 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.704 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:07:40.946 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.731 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:07:40.995 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.861+0000 7fb4f02a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:55.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.861+0000 7fb4f02a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:55.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.861+0000 7fb4f02a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:20:55.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.861+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4e8139770 con 0x7fb4e81430c0 2024-03-25T18:20:55.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.908+0000 7fb4deffd640 1 -- 172.21.15.132:0/73319301 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fb4e8139770 con 0x7fb4e81430c0 2024-03-25T18:20:55.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.908+0000 7fb4deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:55.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.908+0000 7fb4deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:55.909 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:20:55.909 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:55.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 10 librados: watch_flush enter 2024-03-25T18:20:55.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 10 librados: watch_flush exit 2024-03-25T18:20:55.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T18:20:55.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 10 client.4550.objecter successfully canceled tick 2024-03-25T18:20:55.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 >> v1:172.21.15.161:6800/411950978 conn(0x7fb4cc06dc70 legacy=0x7fb4cc070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 10 monclient: shutdown 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.910+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 >> v1:172.21.15.132:6790/0 conn(0x7fb4e81430c0 legacy=0x7fb4e8136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.911+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 shutdown_connections 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.911+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 >> 172.21.15.132:0/73319301 conn(0x7fb4e80b7ea0 msgr2=0x7fb4e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.911+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 shutdown_connections 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.911+0000 7fb4f02a8640 1 -- 172.21.15.132:0/73319301 wait complete. 2024-03-25T18:20:55.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:20:55.911+0000 7fb4f02a8640 1 librados: shutdown 2024-03-25T18:20:55.922 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:02.727 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.846+0000 7f4ea5d9d640 10 monclient: get_monmap_and_config 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.846+0000 7f4ea5d9d640 10 monclient: build_initial_monmap 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.846+0000 7f4ea5d9d640 10 monclient: monmap: 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:02.847318+0000 2024-03-25T18:21:02.847 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:02.847318+0000 2024-03-25T18:21:02.848 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:02.848 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:02.848 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:02.848 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:02.848 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:02.848 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 1 Processor -- start 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 1 -- start start 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient: init 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea00b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea00b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea014be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:02.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea00af940 con 0x7f4ea00b4d60 2024-03-25T18:21:02.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea00b0a70 con 0x7f4ea014be90 2024-03-25T18:21:02.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea00b0d50 con 0x7f4ea00b1730 2024-03-25T18:21:02.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:02.850 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.847+0000 7f4ea5d9d640 10 monclient(hunting): authenticate will time out at 7703.110559s 2024-03-25T18:21:02.851 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.850+0000 7f4e9ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4ea014be90 0x7f4ea014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:48554/0 (socket says 172.21.15.132:48554) 2024-03-25T18:21:02.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.850+0000 7f4e9ffff640 1 -- 172.21.15.132:0/1243078043 learned_addr learned my addr 172.21.15.132:0/1243078043 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:02.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.859+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431616938 0 0) 0x7f4ea00af940 con 0x7f4ea00b4d60 2024-03-25T18:21:02.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.859+0000 7f4e9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.859+0000 7f4e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.859+0000 7f4e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.859+0000 7f4e9e7fc640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:21:02.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.859+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e88003520 con 0x7f4ea00b4d60 2024-03-25T18:21:02.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.862+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939839467 0 0) 0x7f4ea00b0a70 con 0x7f4ea014be90 2024-03-25T18:21:02.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.862+0000 7f4e9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.862+0000 7f4e9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.862+0000 7f4e9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.862+0000 7f4e9e7fc640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:21:02.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.862+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea00af940 con 0x7f4ea014be90 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2298653732 0 0) 0x7f4e88003520 con 0x7f4ea00b4d60 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 >> v1:172.21.15.132:6790/0 conn(0x7f4ea00b1730 legacy=0x7f4ea00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 >> v1:172.21.15.161:6789/0 conn(0x7f4ea014be90 legacy=0x7f4ea014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea00b1030 con 0x7f4ea00b4d60 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:02.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4e90001690 con 0x7f4ea00b4d60 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4e9e7fc640 10 monclient: dump: 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4ea5d9d640 5 monclient: authenticate success, global_id 4551 2024-03-25T18:21:02.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.873+0000 7f4ea5d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e900017f0 con 0x7f4ea00b4d60 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4e9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4e9e7fc640 1 -- 172.21.15.132:0/1243078043 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4e90001c00 con 0x7f4ea00b4d60 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4e9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4e9e7fc640 10 monclient: dump: 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:02.892 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4ea5d9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:02.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.891+0000 7f4ea5d9d640 4 set_mon_vals no callback set 2024-03-25T18:21:02.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 10 monclient: shutdown 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1243078043 >> v1:172.21.15.132:6789/0 conn(0x7f4ea00b4d60 legacy=0x7f4ea013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1243078043 shutdown_connections 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1243078043 >> 172.21.15.132:0/1243078043 conn(0x7f4ea00617a0 msgr2=0x7f4ea0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1243078043 shutdown_connections 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.892+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1243078043 wait complete. 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 10 monclient: build_initial_monmap 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 10 monclient: monmap: 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:02.895 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 1 librados: starting msgr at 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 1 librados: starting objecter 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 1 Processor -- start 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 1 -- start start 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 1 librados: setting wanted keys 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 1 librados: calling monclient init 2024-03-25T18:21:02.896 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.893+0000 7f4ea5d9d640 10 monclient: init 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f4ea00b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f4ea00b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f4ea014be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea013bc50 con 0x7f4ea00b1730 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea0139b60 con 0x7f4ea014be90 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea0134f90 con 0x7f4ea00b4d60 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:02.897 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.894+0000 7f4ea5d9d640 10 monclient(hunting): authenticate will time out at 7703.157313s 2024-03-25T18:21:02.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.903+0000 7f4e9ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4ea014be90 0x7f4ea0134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:48582/0 (socket says 172.21.15.132:48582) 2024-03-25T18:21:02.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.903+0000 7f4e9ffff640 1 -- 172.21.15.132:0/1944222425 learned_addr learned my addr 172.21.15.132:0/1944222425 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:02.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.905+0000 7f4e87fff640 10 client.?.objecter ms_handle_connect 0x7f4ea00b4d60 2024-03-25T18:21:02.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.905+0000 7f4e87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:02.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.907+0000 7f4e87fff640 10 client.?.objecter ms_handle_connect 0x7f4ea00b1730 2024-03-25T18:21:02.908 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.907+0000 7f4e87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:02.909 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.909+0000 7f4e87fff640 10 client.?.objecter ms_handle_connect 0x7f4ea014be90 2024-03-25T18:21:02.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.909+0000 7f4e87fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:02.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.913+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229175729 0 0) 0x7f4ea0134f90 con 0x7f4ea00b4d60 2024-03-25T18:21:02.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.913+0000 7f4e87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.913+0000 7f4e87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.913+0000 7f4e87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.913+0000 7f4e87fff640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:21:02.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.913+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c004eb0 con 0x7f4ea00b4d60 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511493724 0 0) 0x7f4ea013bc50 con 0x7f4ea00b1730 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea0134f90 con 0x7f4ea00b1730 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.922+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998474231 0 0) 0x7f4ea0139b60 con 0x7f4ea014be90 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.923+0000 7f4e87fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.923+0000 7f4e87fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.923+0000 7f4e87fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.923+0000 7f4e87fff640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:21:02.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.923+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea013bc50 con 0x7f4ea014be90 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2691450929 0 0) 0x7f4e8c004eb0 con 0x7f4ea00b4d60 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ea0139b60 con 0x7f4ea00b4d60 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4e94001510 con 0x7f4ea00b4d60 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 10 monclient(hunting): dump: 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.927 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:02.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.926+0000 7f4e87fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:02.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.939+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 777243292 0 0) 0x7f4ea013bc50 con 0x7f4ea014be90 2024-03-25T18:21:02.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e8c004eb0 con 0x7f4ea014be90 2024-03-25T18:21:02.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4e98001b50 con 0x7f4ea014be90 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient(hunting): dump: 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:02.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4150111858 0 0) 0x7f4ea0139b60 con 0x7f4ea00b4d60 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 >> v1:172.21.15.161:6789/0 conn(0x7f4ea014be90 legacy=0x7f4ea0134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 >> v1:172.21.15.132:6789/0 conn(0x7f4ea00b1730 legacy=0x7f4ea0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 monclient: found mon.c 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea0138f60 con 0x7f4ea00b4d60 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4e87fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 10 monclient: _renew_subs 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea0139dd0 con 0x7f4ea00b4d60 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:02.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 10 monclient: _renew_subs 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea0143250 con 0x7f4ea00b4d60 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.940+0000 7f4ea5d9d640 1 librados: init done 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.941+0000 7f4ea5d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.941+0000 7f4ea5d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.941+0000 7f4ea5d9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:02.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:02.941+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea00627b0 con 0x7f4ea00b4d60 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e94001a20 con 0x7f4ea00b4d60 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f4e94003910 con 0x7f4ea00b4d60 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 10 monclient: dump: 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:03.243 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e9dffb640 4 set_mon_vals no callback set 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.242+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f4e940198e0 con 0x7f4ea00b4d60 2024-03-25T18:21:03.244 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.243+0000 7f4e9f7fe640 10 monclient: get_auth_request con 0x7f4e8c06dfb0 auth_method 0 2024-03-25T18:21:03.245 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.243+0000 7f4e87fff640 10 client.4553.objecter ms_handle_connect 0x7f4e8c06dfb0 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.249+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f4e94053580 con 0x7f4ea00b4d60 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.249+0000 7f4e87fff640 10 client.4553.objecter ms_dispatch 0x7f4ea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.249+0000 7f4e87fff640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.249+0000 7f4e87fff640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.249+0000 7f4e87fff640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.250+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e9404f090 con 0x7f4ea00b4d60 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.250+0000 7f4e87fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.250+0000 7f4e87fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:03.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.466+0000 7f4ea5d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.466+0000 7f4ea5d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.466+0000 7f4ea5d9d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:03.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.466+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ea01425a0 con 0x7f4ea00b4d60 2024-03-25T18:21:03.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.500+0000 7f4e87fff640 1 -- 172.21.15.132:0/1944222425 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f4ea01425a0 con 0x7f4ea00b4d60 2024-03-25T18:21:03.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.500+0000 7f4e87fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:03.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.500+0000 7f4e87fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:03.501 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:03.501 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 10 librados: watch_flush enter 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 10 librados: watch_flush exit 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 10 client.4553.objecter successfully canceled tick 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 >> v1:172.21.15.161:6800/411950978 conn(0x7f4e8c06dfb0 legacy=0x7f4e8c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 10 monclient: shutdown 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 >> v1:172.21.15.132:6790/0 conn(0x7f4ea00b4d60 legacy=0x7f4ea0134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 shutdown_connections 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 >> 172.21.15.132:0/1944222425 conn(0x7f4ea00617a0 msgr2=0x7f4ea00b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 shutdown_connections 2024-03-25T18:21:03.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 -- 172.21.15.132:0/1944222425 wait complete. 2024-03-25T18:21:03.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:03.502+0000 7f4ea5d9d640 1 librados: shutdown 2024-03-25T18:21:03.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:10.322 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.433+0000 7f1b13d3d640 10 monclient: get_monmap_and_config 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.433+0000 7f1b13d3d640 10 monclient: build_initial_monmap 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.433+0000 7f1b13d3d640 10 monclient: monmap: 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:10.434956+0000 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:10.434956+0000 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:10.435 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.434+0000 7f1b13d3d640 1 Processor -- start 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.434+0000 7f1b13d3d640 1 -- start start 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.434+0000 7f1b13d3d640 10 monclient: init 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f1b0c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f1b0c0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0c14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c0af940 con 0x7f1b0c0b1730 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c0b0a70 con 0x7f1b0c14be90 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c0b0d50 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:10.438 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.435+0000 7f1b13d3d640 10 monclient(hunting): authenticate will time out at 7710.698293s 2024-03-25T18:21:10.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.445+0000 7f1b11ad9640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f1b0c0b4d60 0x7f1b0c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:40814/0 (socket says 172.21.15.132:40814) 2024-03-25T18:21:10.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.445+0000 7f1b11ad9640 1 -- 172.21.15.132:0/1581389992 learned_addr learned my addr 172.21.15.132:0/1581389992 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:10.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.455+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131649122 0 0) 0x7f1b0c0b0d50 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.455+0000 7f1b10ad7640 10 monclient(hunting): _negotiate 2024-03-25T18:21:10.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.455+0000 7f1b10ad7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:10.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.455+0000 7f1b10ad7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:10.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.455+0000 7f1b10ad7640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:21:10.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.455+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af8003520 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 623970888 0 0) 0x7f1af8003520 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 >> v1:172.21.15.161:6789/0 conn(0x7f1b0c14be90 legacy=0x7f1b0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 >> v1:172.21.15.132:6789/0 conn(0x7f1b0c0b1730 legacy=0x7f1b0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:10.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: _finish_hunting 0 2024-03-25T18:21:10.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 1 monclient: found mon.noname-b 2024-03-25T18:21:10.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c0b1030 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: _finish_auth 0 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: _check_auth_tickets 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b08001690 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b13d3d640 5 monclient: authenticate success, global_id 4556 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b13d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b10ad7640 10 monclient: dump: 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:10.468 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.466+0000 7f1b13d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:10.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b08001860 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b10ad7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b10ad7640 1 -- 172.21.15.132:0/1581389992 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b08001cb0 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b13d3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b13d3d640 4 set_mon_vals no callback set 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b10ad7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b10ad7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.472+0000 7f1b10ad7640 10 monclient: dump: 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:10.474 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:10.475 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:10.475 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:10.475 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:10.475 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.473+0000 7f1b13d3d640 10 monclient: shutdown 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.473+0000 7f1b13d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.473+0000 7f1b13d3d640 1 -- 172.21.15.132:0/1581389992 >> v1:172.21.15.132:6790/0 conn(0x7f1b0c0b4d60 legacy=0x7f1b0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.473+0000 7f1b13d3d640 1 -- 172.21.15.132:0/1581389992 shutdown_connections 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.473+0000 7f1b13d3d640 1 -- 172.21.15.132:0/1581389992 >> 172.21.15.132:0/1581389992 conn(0x7f1b0c0617a0 msgr2=0x7f1b0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.474+0000 7f1b13d3d640 1 -- 172.21.15.132:0/1581389992 shutdown_connections 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.474+0000 7f1b13d3d640 1 -- 172.21.15.132:0/1581389992 wait complete. 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.474+0000 7f1b13d3d640 10 monclient: build_initial_monmap 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.474+0000 7f1b13d3d640 10 monclient: monmap: 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:10.477 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 librados: starting msgr at 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 librados: starting objecter 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 Processor -- start 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 -- start start 2024-03-25T18:21:10.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 librados: setting wanted keys 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 librados: calling monclient init 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient: init 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f1b0c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f1b0c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c145430 con 0x7f1b0c14be90 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c136c20 con 0x7f1b0c0b1730 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b0c143150 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:10.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.475+0000 7f1b13d3d640 10 monclient(hunting): authenticate will time out at 7710.738588s 2024-03-25T18:21:10.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.479+0000 7f1b122da640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f1b0c14be90 0x7f1b0c1d77b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:36338/0 (socket says 172.21.15.132:36338) 2024-03-25T18:21:10.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.479+0000 7f1b122da640 1 -- 172.21.15.132:0/283304135 learned_addr learned my addr 172.21.15.132:0/283304135 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:10.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.480+0000 7f1b027fc640 10 client.?.objecter ms_handle_connect 0x7f1b0c14be90 2024-03-25T18:21:10.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.480+0000 7f1b027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:10.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.485+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3969250965 0 0) 0x7f1b0c145430 con 0x7f1b0c14be90 2024-03-25T18:21:10.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.485+0000 7f1b027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:10.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.485+0000 7f1b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:10.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.485+0000 7f1b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:10.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.485+0000 7f1b027fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:21:10.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.485+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af0004eb0 con 0x7f1b0c14be90 2024-03-25T18:21:10.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.488+0000 7f1b027fc640 10 client.?.objecter ms_handle_connect 0x7f1b0c0b4d60 2024-03-25T18:21:10.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.488+0000 7f1b027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:10.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.491+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2405353682 0 0) 0x7f1b0c143150 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.491+0000 7f1b027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:10.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.491+0000 7f1b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:10.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.491+0000 7f1b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:10.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.491+0000 7f1b027fc640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:21:10.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.491+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c145430 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.492+0000 7f1b027fc640 10 client.?.objecter ms_handle_connect 0x7f1b0c0b1730 2024-03-25T18:21:10.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.492+0000 7f1b027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.494+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2617362725 0 0) 0x7f1af0004eb0 con 0x7f1b0c14be90 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.495+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b0c143150 con 0x7f1b0c14be90 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.495+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b04001c90 con 0x7f1b0c14be90 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.495+0000 7f1b027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.495+0000 7f1b027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.495+0000 7f1b027fc640 10 monclient(hunting): dump: 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:10.496 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:10.497 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:10.497 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:10.497 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:10.497 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.495+0000 7f1b027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.500+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416869310 0 0) 0x7f1b0c145430 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.500+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af0004eb0 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1afc001510 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): dump: 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.503 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982773587 0 0) 0x7f1b0c136c20 con 0x7f1b0c0b1730 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b0c145430 con 0x7f1b0c0b1730 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830347333 0 0) 0x7f1b0c143150 con 0x7f1b0c14be90 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6790/0 conn(0x7f1b0c0b4d60 legacy=0x7f1b0c142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.161:6789/0 conn(0x7f1b0c0b1730 legacy=0x7f1b0c143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 monclient: found mon.a 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b0c1432e0 con 0x7f1b0c14be90 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 10 monclient: _renew_subs 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b0c13ed30 con 0x7f1b0c14be90 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 10 monclient: _renew_subs 2024-03-25T18:21:10.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:10.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b0c1d80a0 con 0x7f1b0c14be90 2024-03-25T18:21:10.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.501+0000 7f1b13d3d640 1 librados: init done 2024-03-25T18:21:10.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.502+0000 7f1b13d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:10.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.502+0000 7f1b13d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:10.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.502+0000 7f1b13d3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:10.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.502+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c1d8350 con 0x7f1b0c14be90 2024-03-25T18:21:10.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b122da640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6789/0 conn(0x7f1b0c14be90 legacy=0x7f1b0c1d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:21:10.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b122da640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6789/0 conn(0x7f1b0c14be90 legacy=0x7f1b0c1d77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:21:10.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b122da640 1 --1- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6789/0 conn(0x7f1b0c14be90 0x7f1b0c1d77b0 :-1 s=OPENED pgs=55726 cs=1 l=1).handle_message read tag failed 2024-03-25T18:21:10.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b122da640 1 --1- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6789/0 conn(0x7f1b0c14be90 0x7f1b0c1d77b0 :-1 s=OPENED pgs=55726 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:21:10.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.132:6789/0 2024-03-25T18:21:10.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6789/0 conn(0x7f1b0c14be90 legacy=0x7f1b0c1d77b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient(hunting): picked mon.c con 0x7f1b0c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient(hunting): picked mon.a con 0x7f1af000a300 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient(hunting): picked mon.b con 0x7f1b0c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1b0c143150 con 0x7f1af000a300 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1b0c136c20 con 0x7f1b0c0b1730 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f1af0013df0 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.508+0000 7f1b027fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:10.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.511+0000 7f1b027fc640 10 client.4557.objecter ms_handle_connect 0x7f1b0c0b4d60 2024-03-25T18:21:10.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.511+0000 7f1b027fc640 10 client.4557.objecter resend_mon_ops 2024-03-25T18:21:10.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.517+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193118279 0 0) 0x7f1af0013df0 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.517+0000 7f1b027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:10.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.517+0000 7f1b027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:10.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.517+0000 7f1b027fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:21:10.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.517+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f1af0013f80 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.523+0000 7f1b027fc640 10 client.4557.objecter ms_handle_connect 0x7f1b0c0b1730 2024-03-25T18:21:10.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.523+0000 7f1b027fc640 10 client.4557.objecter resend_mon_ops 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 54844411 0 0) 0x7f1af0013f80 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.161:6789/0 conn(0x7f1b0c0b1730 legacy=0x7f1af0010190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6789/0 conn(0x7f1af000a300 legacy=0x7f1af000c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 monclient: found mon.c 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f1af00151d0 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b0c143150 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.533 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1b08001d40 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.531+0000 7f1b027fc640 10 monclient: dump: 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:10.534 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:10.535 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:10.535 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:10.535 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:10.825 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.824+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b08001210 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.826 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.824+0000 7f1b027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:10.826 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.824+0000 7f1b03fff640 4 set_mon_vals no callback set 2024-03-25T18:21:10.826 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.825+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f1b08001400 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.826+0000 7f1b112d8640 10 monclient: get_auth_request con 0x7f1b0c14be90 auth_method 0 2024-03-25T18:21:10.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.826+0000 7f1b027fc640 10 client.4557.objecter ms_handle_connect 0x7f1b0c14be90 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f1b08054000 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 10 client.4557.objecter ms_dispatch 0x7f1b0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b0804fb10 con 0x7f1b0c0b4d60 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:10.836 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:10.835+0000 7f1b027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:11.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.085+0000 7f1b13d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.085+0000 7f1b13d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.085+0000 7f1b13d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:11.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.085+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b0c13f0d0 con 0x7f1b0c0b4d60 2024-03-25T18:21:11.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.119+0000 7f1b027fc640 1 -- 172.21.15.132:0/283304135 <== mon.2 v1:172.21.15.132:6790/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f1b08051a40 con 0x7f1b0c0b4d60 2024-03-25T18:21:11.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.119+0000 7f1b027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.119+0000 7f1b027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:11.120 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:11.120 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:11.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.121+0000 7f1b13d3d640 10 librados: watch_flush enter 2024-03-25T18:21:11.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 10 librados: watch_flush exit 2024-03-25T18:21:11.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 10 client.4557.objecter successfully canceled tick 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.161:6800/411950978 conn(0x7f1b0c14be90 legacy=0x7f1af007bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 10 monclient: shutdown 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 >> v1:172.21.15.132:6790/0 conn(0x7f1b0c0b4d60 legacy=0x7f1af0006750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 shutdown_connections 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 >> 172.21.15.132:0/283304135 conn(0x7f1b0c0617a0 msgr2=0x7f1b0c0b6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 shutdown_connections 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 -- 172.21.15.132:0/283304135 wait complete. 2024-03-25T18:21:11.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:11.122+0000 7f1b13d3d640 1 librados: shutdown 2024-03-25T18:21:11.134 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:15.401 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:00.664 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.404 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:00.667 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.438 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:00.702 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.455 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:00.697 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.461 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:00.725 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.467 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:00.708 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.470 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:00.734 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.503 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:00.744 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:15.513 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:00.777 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:17.940 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:18.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.055+0000 7f2b16de8640 10 monclient: get_monmap_and_config 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.055+0000 7f2b16de8640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.055+0000 7f2b16de8640 10 monclient: monmap: 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:18.056848+0000 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:18.056848+0000 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:18.057 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.056+0000 7f2b16de8640 1 Processor -- start 2024-03-25T18:21:18.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.056+0000 7f2b16de8640 1 -- start start 2024-03-25T18:21:18.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.056+0000 7f2b16de8640 10 monclient: init 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient(hunting): picked mon.noname-a con 0x7f2b100a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient(hunting): picked mon.noname-b con 0x7f2b1013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient(hunting): picked mon.noname-c con 0x7f2b101430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b10142520 con 0x7f2b100a98f0 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100b28a0 con 0x7f2b101430c0 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100b2a30 con 0x7f2b1013f710 2024-03-25T18:21:18.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.060 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.057+0000 7f2b16de8640 10 monclient(hunting): authenticate will time out at 7718.320625s 2024-03-25T18:21:18.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.069+0000 7f2b0ffff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f2b1013f710 0x7f2b101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:40300/0 (socket says 172.21.15.132:40300) 2024-03-25T18:21:18.070 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.069+0000 7f2b0ffff640 1 -- 172.21.15.132:0/1336215920 learned_addr learned my addr 172.21.15.132:0/1336215920 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:18.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.083+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624013703 0 0) 0x7f2b100b28a0 con 0x7f2b101430c0 2024-03-25T18:21:18.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.083+0000 7f2b0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.083+0000 7f2b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.083+0000 7f2b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.083+0000 7f2b0f7fe640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:21:18.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.083+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b00003520 con 0x7f2b101430c0 2024-03-25T18:21:18.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.091+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346649219 0 0) 0x7f2b100b2a30 con 0x7f2b1013f710 2024-03-25T18:21:18.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.091+0000 7f2b0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.091+0000 7f2b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.091+0000 7f2b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.091+0000 7f2b0f7fe640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:21:18.092 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.091+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b100b28a0 con 0x7f2b1013f710 2024-03-25T18:21:18.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697977395 0 0) 0x7f2b10142520 con 0x7f2b100a98f0 2024-03-25T18:21:18.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.098 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b100b2a30 con 0x7f2b100a98f0 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941303079 0 0) 0x7f2b00003520 con 0x7f2b101430c0 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 >> v1:172.21.15.132:6790/0 conn(0x7f2b1013f710 legacy=0x7f2b101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 >> v1:172.21.15.132:6789/0 conn(0x7f2b100a98f0 legacy=0x7f2b100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b100b44c0 con 0x7f2b101430c0 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b16de8640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:21:18.099 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.096+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2b08001a10 con 0x7f2b101430c0 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.097+0000 7f2b0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.097+0000 7f2b0f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.097+0000 7f2b0f7fe640 10 monclient: dump: 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.097+0000 7f2b16de8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b08001bf0 con 0x7f2b101430c0 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b0f7fe640 1 -- 172.21.15.132:0/1336215920 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2b08002000 con 0x7f2b101430c0 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b16de8640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b16de8640 4 set_mon_vals no callback set 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b0f7fe640 10 monclient: dump: 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:18.102 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:18.103 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b16de8640 10 monclient: shutdown 2024-03-25T18:21:18.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b16de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:18.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.101+0000 7f2b16de8640 1 -- 172.21.15.132:0/1336215920 >> v1:172.21.15.161:6789/0 conn(0x7f2b101430c0 legacy=0x7f2b1014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.102+0000 7f2b16de8640 1 -- 172.21.15.132:0/1336215920 shutdown_connections 2024-03-25T18:21:18.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.102+0000 7f2b16de8640 1 -- 172.21.15.132:0/1336215920 >> 172.21.15.132:0/1336215920 conn(0x7f2b100b7ea0 msgr2=0x7f2b100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.102+0000 7f2b16de8640 1 -- 172.21.15.132:0/1336215920 shutdown_connections 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.102+0000 7f2b16de8640 1 -- 172.21.15.132:0/1336215920 wait complete. 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.102+0000 7f2b16de8640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.102+0000 7f2b16de8640 10 monclient: monmap: 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 1 librados: starting msgr at 2024-03-25T18:21:18.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 1 librados: starting objecter 2024-03-25T18:21:18.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 1 Processor -- start 2024-03-25T18:21:18.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 1 -- start start 2024-03-25T18:21:18.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 1 librados: setting wanted keys 2024-03-25T18:21:18.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 1 librados: calling monclient init 2024-03-25T18:21:18.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.103+0000 7f2b16de8640 10 monclient: init 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient(hunting): picked mon.noname-b con 0x7f2b100a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient(hunting): picked mon.noname-a con 0x7f2b1013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient(hunting): picked mon.noname-c con 0x7f2b101430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b101daca0 con 0x7f2b1013f710 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b10138ab0 con 0x7f2b100a98f0 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100aba90 con 0x7f2b101430c0 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.104+0000 7f2b16de8640 10 monclient(hunting): authenticate will time out at 7718.367337s 2024-03-25T18:21:18.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.122+0000 7f2b15385640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f2b101430c0 0x7f2b101d47b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:40360/0 (socket says 172.21.15.132:40360) 2024-03-25T18:21:18.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.122+0000 7f2b15385640 1 -- 172.21.15.132:0/472662900 learned_addr learned my addr 172.21.15.132:0/472662900 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:18.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.124+0000 7f2b0d7fa640 10 client.?.objecter ms_handle_connect 0x7f2b101430c0 2024-03-25T18:21:18.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.124+0000 7f2b0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.124+0000 7f2b0d7fa640 10 client.?.objecter ms_handle_connect 0x7f2b100a98f0 2024-03-25T18:21:18.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.124+0000 7f2b0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.132+0000 7f2b0d7fa640 10 client.?.objecter ms_handle_connect 0x7f2b1013f710 2024-03-25T18:21:18.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.132+0000 7f2b0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.135+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557494908 0 0) 0x7f2b10138ab0 con 0x7f2b100a98f0 2024-03-25T18:21:18.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.135+0000 7f2b0d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.135+0000 7f2b0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.135+0000 7f2b0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.135+0000 7f2b0d7fa640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:21:18.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.135+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af4004eb0 con 0x7f2b100a98f0 2024-03-25T18:21:18.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.137+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545808757 0 0) 0x7f2b100aba90 con 0x7f2b101430c0 2024-03-25T18:21:18.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.137+0000 7f2b0d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.137+0000 7f2b0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.137+0000 7f2b0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.137+0000 7f2b0d7fa640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:21:18.138 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.137+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b10138ab0 con 0x7f2b101430c0 2024-03-25T18:21:18.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.150+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265603090 0 0) 0x7f2b101daca0 con 0x7f2b1013f710 2024-03-25T18:21:18.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.150+0000 7f2b0d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.150+0000 7f2b0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.150+0000 7f2b0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.150+0000 7f2b0d7fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:21:18.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.150+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b100aba90 con 0x7f2b1013f710 2024-03-25T18:21:18.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2291768849 0 0) 0x7f2af4004eb0 con 0x7f2b100a98f0 2024-03-25T18:21:18.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b101daca0 con 0x7f2b100a98f0 2024-03-25T18:21:18.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2b040018f0 con 0x7f2b100a98f0 2024-03-25T18:21:18.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:18.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.162 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.160+0000 7f2b0d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:18.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.168+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2891154769 0 0) 0x7f2b100aba90 con 0x7f2b1013f710 2024-03-25T18:21:18.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.168+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af4004eb0 con 0x7f2b1013f710 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.168+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2afc0018b0 con 0x7f2b1013f710 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.168+0000 7f2b0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.168+0000 7f2b0d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.168+0000 7f2b0d7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:18.170 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.179+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724168164 0 0) 0x7f2b10138ab0 con 0x7f2b101430c0 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.179+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b100aba90 con 0x7f2b101430c0 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.179+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2b08001f00 con 0x7f2b101430c0 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.179+0000 7f2b0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.179+0000 7f2b0d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.179+0000 7f2b0d7fa640 10 monclient(hunting): dump: 2024-03-25T18:21:18.180 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:18.181 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2098881992 0 0) 0x7f2b101daca0 con 0x7f2b100a98f0 2024-03-25T18:21:18.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 >> v1:172.21.15.132:6790/0 conn(0x7f2b101430c0 legacy=0x7f2b101d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 >> v1:172.21.15.132:6789/0 conn(0x7f2b1013f710 legacy=0x7f2b101db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 1 monclient: found mon.b 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b101d41e0 con 0x7f2b100a98f0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 10 monclient: _renew_subs 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b101db840 con 0x7f2b100a98f0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 10 monclient: _renew_subs 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:18.184 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b10138f00 con 0x7f2b100a98f0 2024-03-25T18:21:18.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.182+0000 7f2b16de8640 1 librados: init done 2024-03-25T18:21:18.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.183+0000 7f2b16de8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.183+0000 7f2b16de8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.183+0000 7f2b16de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:18.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.183+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b101dba80 con 0x7f2b100a98f0 2024-03-25T18:21:18.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.393+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b04003610 con 0x7f2b100a98f0 2024-03-25T18:21:18.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.393+0000 7f2b0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:18.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.393+0000 7f2b0effd640 4 set_mon_vals no callback set 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.452+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2b04003920 con 0x7f2b100a98f0 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.452+0000 7f2b0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.452+0000 7f2b0d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.452+0000 7f2b0d7fa640 10 monclient: dump: 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:18.454 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:18.455 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:18.455 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:18.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.453+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f2b040198f0 con 0x7f2b100a98f0 2024-03-25T18:21:18.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.454+0000 7f2b0ffff640 10 monclient: get_auth_request con 0x7f2af406e0f0 auth_method 0 2024-03-25T18:21:18.455 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.454+0000 7f2b0d7fa640 10 client.4573.objecter ms_handle_connect 0x7f2af406e0f0 2024-03-25T18:21:18.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.456+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f2b040536a0 con 0x7f2b100a98f0 2024-03-25T18:21:18.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.456+0000 7f2b0d7fa640 10 client.4573.objecter ms_dispatch 0x7f2b10000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:18.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.456+0000 7f2b0d7fa640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:18.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.456+0000 7f2b0d7fa640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:18.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.456+0000 7f2b0d7fa640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-25T18:21:18.459 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.458+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b0404efe0 con 0x7f2b100a98f0 2024-03-25T18:21:18.459 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.458+0000 7f2b0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.459 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.458+0000 7f2b0d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:18.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.714+0000 7f2b16de8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:18.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.714+0000 7f2b16de8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:18.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.714+0000 7f2b16de8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:18.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.714+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b100627b0 con 0x7f2b100a98f0 2024-03-25T18:21:18.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.751+0000 7f2b0d7fa640 1 -- 172.21.15.132:0/472662900 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f2b040510e0 con 0x7f2b100a98f0 2024-03-25T18:21:18.753 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.751+0000 7f2b0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:18.753 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.751+0000 7f2b0d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:18.753 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:18.753 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 10 librados: watch_flush enter 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 10 librados: watch_flush exit 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 10 client.4573.objecter successfully canceled tick 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 >> v1:172.21.15.161:6800/411950978 conn(0x7f2af406e0f0 legacy=0x7f2af40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 10 monclient: shutdown 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 >> v1:172.21.15.161:6789/0 conn(0x7f2b100a98f0 legacy=0x7f2b101383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 shutdown_connections 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 >> 172.21.15.132:0/472662900 conn(0x7f2b100b7ea0 msgr2=0x7f2b10074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 shutdown_connections 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 -- 172.21.15.132:0/472662900 wait complete. 2024-03-25T18:21:18.756 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:18.754+0000 7f2b16de8640 1 librados: shutdown 2024-03-25T18:21:18.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:25.574 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:25.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.690+0000 7fe7b8d16640 10 monclient: get_monmap_and_config 2024-03-25T18:21:25.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.690+0000 7fe7b8d16640 10 monclient: build_initial_monmap 2024-03-25T18:21:25.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.690+0000 7fe7b8d16640 10 monclient: monmap: 2024-03-25T18:21:25.691 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:25.691 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:25.691 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:25.691360+0000 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:25.691360+0000 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:25.692 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:25.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.691+0000 7fe7b8d16640 1 Processor -- start 2024-03-25T18:21:25.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.691+0000 7fe7b8d16640 1 -- start start 2024-03-25T18:21:25.693 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.691+0000 7fe7b8d16640 10 monclient: init 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.691+0000 7fe7b8d16640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.691+0000 7fe7b8d16640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b40a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b413f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b41430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b4142520 con 0x7fe7b41430c0 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40b28a0 con 0x7fe7b40a98f0 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b40b2a30 con 0x7fe7b413f710 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:25.694 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.692+0000 7fe7b8d16640 10 monclient(hunting): authenticate will time out at 7725.955222s 2024-03-25T18:21:25.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.701+0000 7fe7b259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe7b40a98f0 0x7fe7b40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:52672/0 (socket says 172.21.15.132:52672) 2024-03-25T18:21:25.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.701+0000 7fe7b259c640 1 -- 172.21.15.132:0/3599032929 learned_addr learned my addr 172.21.15.132:0/3599032929 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:25.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.715+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934797210 0 0) 0x7fe7b40b2a30 con 0x7fe7b413f710 2024-03-25T18:21:25.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.715+0000 7fe7b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.715+0000 7fe7b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.715+0000 7fe7b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.715+0000 7fe7b1d9b640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:21:25.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.715+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7a0003520 con 0x7fe7b413f710 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.716+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262112451 0 0) 0x7fe7b40b28a0 con 0x7fe7b40a98f0 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.716+0000 7fe7b1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.716+0000 7fe7b1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.716+0000 7fe7b1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.716+0000 7fe7b1d9b640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:21:25.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.716+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b40b2a30 con 0x7fe7b40a98f0 2024-03-25T18:21:25.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4294448021 0 0) 0x7fe7a0003520 con 0x7fe7b413f710 2024-03-25T18:21:25.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 >> v1:172.21.15.161:6789/0 conn(0x7fe7b40a98f0 legacy=0x7fe7b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 >> v1:172.21.15.132:6789/0 conn(0x7fe7b41430c0 legacy=0x7fe7b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:25.723 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b40b44c0 con 0x7fe7b413f710 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe79c001690 con 0x7fe7b413f710 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b8d16640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b8d16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b1d9b640 10 monclient: dump: 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:25.724 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:25.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.722+0000 7fe7b8d16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:25.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe79c0017f0 con 0x7fe7b413f710 2024-03-25T18:21:25.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:25.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b1d9b640 1 -- 172.21.15.132:0/3599032929 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe79c001c00 con 0x7fe7b413f710 2024-03-25T18:21:25.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b8d16640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:25.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b8d16640 4 set_mon_vals no callback set 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.744+0000 7fe7b1d9b640 10 monclient: dump: 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:25.746 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:25.747 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 10 monclient: shutdown 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 1 -- 172.21.15.132:0/3599032929 >> v1:172.21.15.132:6790/0 conn(0x7fe7b413f710 legacy=0x7fe7b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 1 -- 172.21.15.132:0/3599032929 shutdown_connections 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 1 -- 172.21.15.132:0/3599032929 >> 172.21.15.132:0/3599032929 conn(0x7fe7b40b7ea0 msgr2=0x7fe7b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 1 -- 172.21.15.132:0/3599032929 shutdown_connections 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.745+0000 7fe7b8d16640 1 -- 172.21.15.132:0/3599032929 wait complete. 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.746+0000 7fe7b8d16640 10 monclient: build_initial_monmap 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.746+0000 7fe7b8d16640 10 monclient: monmap: 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:25.748 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.746+0000 7fe7b8d16640 1 librados: starting msgr at 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.746+0000 7fe7b8d16640 1 librados: starting objecter 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.746+0000 7fe7b8d16640 1 Processor -- start 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 1 -- start start 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 1 librados: setting wanted keys 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 1 librados: calling monclient init 2024-03-25T18:21:25.749 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient: init 2024-03-25T18:21:25.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:25.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:25.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient(hunting): picked mon.noname-b con 0x7fe7b40a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:25.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient(hunting): picked mon.noname-a con 0x7fe7b413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:25.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient(hunting): picked mon.noname-c con 0x7fe7b41430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:25.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b413be70 con 0x7fe7b413f710 2024-03-25T18:21:25.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b4131660 con 0x7fe7b40a98f0 2024-03-25T18:21:25.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b41317f0 con 0x7fe7b41430c0 2024-03-25T18:21:25.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:25.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.747+0000 7fe7b8d16640 10 monclient(hunting): authenticate will time out at 7726.010594s 2024-03-25T18:21:25.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.753+0000 7fe7b259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe7b40a98f0 0x7fe7b4130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:52718/0 (socket says 172.21.15.132:52718) 2024-03-25T18:21:25.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.753+0000 7fe7b259c640 1 -- 172.21.15.132:0/1471143453 learned_addr learned my addr 172.21.15.132:0/1471143453 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:25.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.754+0000 7fe7aaffd640 10 client.?.objecter ms_handle_connect 0x7fe7b40a98f0 2024-03-25T18:21:25.755 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.754+0000 7fe7aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:25.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.767+0000 7fe7aaffd640 10 client.?.objecter ms_handle_connect 0x7fe7b413f710 2024-03-25T18:21:25.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.767+0000 7fe7aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:25.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.772+0000 7fe7aaffd640 10 client.?.objecter ms_handle_connect 0x7fe7b41430c0 2024-03-25T18:21:25.773 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.772+0000 7fe7aaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:25.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.775+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627107409 0 0) 0x7fe7b4131660 con 0x7fe7b40a98f0 2024-03-25T18:21:25.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.775+0000 7fe7aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.775+0000 7fe7aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.775+0000 7fe7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.775+0000 7fe7aaffd640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:21:25.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.775+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe794004eb0 con 0x7fe7b40a98f0 2024-03-25T18:21:25.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.788+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434006219 0 0) 0x7fe7b413be70 con 0x7fe7b413f710 2024-03-25T18:21:25.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.788+0000 7fe7aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.788+0000 7fe7aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.788+0000 7fe7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.788+0000 7fe7aaffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:21:25.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.788+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7b4131660 con 0x7fe7b413f710 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2418360904 0 0) 0x7fe794004eb0 con 0x7fe7b40a98f0 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b413be70 con 0x7fe7b40a98f0 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe7a40019f0 con 0x7fe7b40a98f0 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 10 monclient(hunting): dump: 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:25.795 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:25.796 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:25.796 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:25.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.793+0000 7fe7aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:25.796 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.795+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493608622 0 0) 0x7fe7b41317f0 con 0x7fe7b41430c0 2024-03-25T18:21:25.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.795+0000 7fe7aaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:25.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.795+0000 7fe7aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:25.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.795+0000 7fe7aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:25.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.795+0000 7fe7aaffd640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:21:25.797 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.795+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe794004eb0 con 0x7fe7b41430c0 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.808+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2137368847 0 0) 0x7fe7b4131660 con 0x7fe7b413f710 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.808+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe7b41317f0 con 0x7fe7b413f710 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.808+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe79c001b00 con 0x7fe7b413f710 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.808+0000 7fe7aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.808+0000 7fe7aaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.808+0000 7fe7aaffd640 10 monclient(hunting): dump: 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.810 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:25.811 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:25.811 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:25.811 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:25.811 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2186307096 0 0) 0x7fe7b413be70 con 0x7fe7b40a98f0 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 >> v1:172.21.15.132:6790/0 conn(0x7fe7b41430c0 legacy=0x7fe7b41e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 >> v1:172.21.15.132:6789/0 conn(0x7fe7b413f710 legacy=0x7fe7b41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 1 monclient: found mon.b 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4131980 con 0x7fe7b40a98f0 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:25.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 10 monclient: _renew_subs 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b4135c60 con 0x7fe7b40a98f0 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 10 monclient: _renew_subs 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b40b7770 con 0x7fe7b40a98f0 2024-03-25T18:21:25.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.820+0000 7fe7b8d16640 1 librados: init done 2024-03-25T18:21:25.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.822+0000 7fe7b8d16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:25.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.822+0000 7fe7b8d16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:25.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.822+0000 7fe7b8d16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:25.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:25.822+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b413c080 con 0x7fe7b40a98f0 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7a40047e0 con 0x7fe7b40a98f0 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe7a4004af0 con 0x7fe7b40a98f0 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7aaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7aaffd640 10 monclient: dump: 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.100 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:26.101 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:26.101 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:26.101 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:26.101 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:26.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.098+0000 7fe7b159a640 4 set_mon_vals no callback set 2024-03-25T18:21:26.101 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.099+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fe7a401aac0 con 0x7fe7b40a98f0 2024-03-25T18:21:26.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.100+0000 7fe7abfff640 10 monclient: get_auth_request con 0x7fe79406dfb0 auth_method 0 2024-03-25T18:21:26.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.100+0000 7fe7aaffd640 10 client.4579.objecter ms_handle_connect 0x7fe79406dfb0 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fe7a40548d0 con 0x7fe7b40a98f0 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 10 client.4579.objecter ms_dispatch 0x7fe7b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7a4050210 con 0x7fe7b40a98f0 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.104+0000 7fe7aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:26.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.333+0000 7fe7b8d16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:26.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.333+0000 7fe7b8d16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:26.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.333+0000 7fe7b8d16640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:21:26.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.333+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7b41371d0 con 0x7fe7b40a98f0 2024-03-25T18:21:26.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.374+0000 7fe7aaffd640 1 -- 172.21.15.132:0/1471143453 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fe7a4052310 con 0x7fe7b40a98f0 2024-03-25T18:21:26.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.374+0000 7fe7aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:26.375 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.374+0000 7fe7aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:26.375 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:26.376 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:26.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.376+0000 7fe7b8d16640 10 librados: watch_flush enter 2024-03-25T18:21:26.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.376+0000 7fe7b8d16640 10 librados: watch_flush exit 2024-03-25T18:21:26.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.376+0000 7fe7b8d16640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T18:21:26.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.376+0000 7fe7b8d16640 10 client.4579.objecter successfully canceled tick 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 >> v1:172.21.15.161:6800/411950978 conn(0x7fe79406dfb0 legacy=0x7fe794070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 10 monclient: shutdown 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 >> v1:172.21.15.161:6789/0 conn(0x7fe7b40a98f0 legacy=0x7fe7b4130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 shutdown_connections 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 >> 172.21.15.132:0/1471143453 conn(0x7fe7b40b7ea0 msgr2=0x7fe7b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 shutdown_connections 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 -- 172.21.15.132:0/1471143453 wait complete. 2024-03-25T18:21:26.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:26.377+0000 7fe7b8d16640 1 librados: shutdown 2024-03-25T18:21:26.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:33.195 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.300+0000 7f389794b640 10 monclient: get_monmap_and_config 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.300+0000 7f389794b640 10 monclient: build_initial_monmap 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.300+0000 7f389794b640 10 monclient: monmap: 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:33.301392+0000 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:33.301392+0000 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:33.301 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.302 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:33.302 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:33.302 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:33.302 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.300+0000 7f389794b640 1 Processor -- start 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.301+0000 7f389794b640 1 -- start start 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.301+0000 7f389794b640 10 monclient: init 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.301+0000 7f389794b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.301+0000 7f389794b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.301+0000 7f389794b640 10 monclient(hunting): picked mon.noname-b con 0x7f38900a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 10 monclient(hunting): picked mon.noname-a con 0x7f389013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 10 monclient(hunting): picked mon.noname-c con 0x7f38901430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3890142520 con 0x7f389013f710 2024-03-25T18:21:33.303 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900b28a0 con 0x7f38901430c0 2024-03-25T18:21:33.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900b2a30 con 0x7f38900a98f0 2024-03-25T18:21:33.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:33.304 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.302+0000 7f389794b640 10 monclient(hunting): authenticate will time out at 7733.564894s 2024-03-25T18:21:33.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.308+0000 7f3894ee6640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f389013f710 0x7f38901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39612/0 (socket says 172.21.15.132:39612) 2024-03-25T18:21:33.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.308+0000 7f3894ee6640 1 -- 172.21.15.132:0/3074128257 learned_addr learned my addr 172.21.15.132:0/3074128257 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:33.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.315+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163120709 0 0) 0x7f3890142520 con 0x7f389013f710 2024-03-25T18:21:33.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.316+0000 7f3887fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:33.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.316+0000 7f3887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:33.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.316+0000 7f3887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:33.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.316+0000 7f3887fff640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:21:33.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.316+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f387c003520 con 0x7f389013f710 2024-03-25T18:21:33.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.326+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089900576 0 0) 0x7f38900b2a30 con 0x7f38900a98f0 2024-03-25T18:21:33.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.326+0000 7f3887fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:33.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.326+0000 7f3887fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:33.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.326+0000 7f3887fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:33.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.326+0000 7f3887fff640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:21:33.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.327+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3890142520 con 0x7f38900a98f0 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107710777 0 0) 0x7f387c003520 con 0x7f389013f710 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 >> v1:172.21.15.132:6790/0 conn(0x7f38900a98f0 legacy=0x7f38900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 >> v1:172.21.15.161:6789/0 conn(0x7f38901430c0 legacy=0x7f389014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 1 monclient: found mon.noname-a 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38900b44c0 con 0x7f389013f710 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3880001690 con 0x7f389013f710 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.330 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f3887fff640 10 monclient: dump: 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f389794b640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:21:33.331 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.328+0000 7f389794b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:33.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38800017f0 con 0x7f389013f710 2024-03-25T18:21:33.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f3887fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:33.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f3887fff640 1 -- 172.21.15.132:0/3074128257 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3880001c00 con 0x7f389013f710 2024-03-25T18:21:33.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f3887fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f3887fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f3887fff640 10 monclient: dump: 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f389794b640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:33.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.332+0000 7f389794b640 4 set_mon_vals no callback set 2024-03-25T18:21:33.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 10 monclient: shutdown 2024-03-25T18:21:33.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:33.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 1 -- 172.21.15.132:0/3074128257 >> v1:172.21.15.132:6789/0 conn(0x7f389013f710 legacy=0x7f38901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 1 -- 172.21.15.132:0/3074128257 shutdown_connections 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 1 -- 172.21.15.132:0/3074128257 >> 172.21.15.132:0/3074128257 conn(0x7f38900b7ea0 msgr2=0x7f38900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 1 -- 172.21.15.132:0/3074128257 shutdown_connections 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.333+0000 7f389794b640 1 -- 172.21.15.132:0/3074128257 wait complete. 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 10 monclient: build_initial_monmap 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 10 monclient: monmap: 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 1 librados: starting msgr at 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 1 librados: starting objecter 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 1 Processor -- start 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 1 -- start start 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 1 librados: setting wanted keys 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 1 librados: calling monclient init 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.334+0000 7f389794b640 10 monclient: init 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient(hunting): picked mon.noname-b con 0x7f38900a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient(hunting): picked mon.noname-a con 0x7f389013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient(hunting): picked mon.noname-c con 0x7f38901430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900abac0 con 0x7f389013f710 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900abda0 con 0x7f38900a98f0 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38901380e0 con 0x7f38901430c0 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:33.338 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.335+0000 7f389794b640 10 monclient(hunting): authenticate will time out at 7733.597925s 2024-03-25T18:21:33.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.343+0000 7f3894ee6640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f389013f710 0x7f3890138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39642/0 (socket says 172.21.15.132:39642) 2024-03-25T18:21:33.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.343+0000 7f3894ee6640 1 -- 172.21.15.132:0/3839018646 learned_addr learned my addr 172.21.15.132:0/3839018646 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:33.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.346+0000 7f3885ffb640 10 client.?.objecter ms_handle_connect 0x7f389013f710 2024-03-25T18:21:33.347 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.346+0000 7f3885ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:33.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.350+0000 7f3885ffb640 10 client.?.objecter ms_handle_connect 0x7f38901430c0 2024-03-25T18:21:33.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.350+0000 7f3885ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503905318 0 0) 0x7f38901380e0 con 0x7f38901430c0 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3874004eb0 con 0x7f38901430c0 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 10 client.?.objecter ms_handle_connect 0x7f38900a98f0 2024-03-25T18:21:33.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.358+0000 7f3885ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:33.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.367+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264396320 0 0) 0x7f38900abac0 con 0x7f389013f710 2024-03-25T18:21:33.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.367+0000 7f3885ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:33.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.367+0000 7f3885ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:33.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.367+0000 7f3885ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:33.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.367+0000 7f3885ffb640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:21:33.368 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.367+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38901380e0 con 0x7f389013f710 2024-03-25T18:21:33.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.369+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505179620 0 0) 0x7f38900abda0 con 0x7f38900a98f0 2024-03-25T18:21:33.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.369+0000 7f3885ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:33.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.369+0000 7f3885ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:33.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.369+0000 7f3885ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:33.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.369+0000 7f3885ffb640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:21:33.370 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.369+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38900abac0 con 0x7f38900a98f0 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.376+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 841608290 0 0) 0x7f3874004eb0 con 0x7f38901430c0 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.377+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38900abda0 con 0x7f38901430c0 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.377+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3888001b90 con 0x7f38901430c0 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.377+0000 7f3885ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.377+0000 7f3885ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.377+0000 7f3885ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.378 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.379 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.379 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:33.379 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:33.379 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:33.379 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.377+0000 7f3885ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.380+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1726518060 0 0) 0x7f38900abac0 con 0x7f38900a98f0 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.380+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3874004eb0 con 0x7f38900a98f0 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.380+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f388c001930 con 0x7f38900a98f0 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.380+0000 7f3885ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.380+0000 7f3885ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.380+0000 7f3885ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:33.381 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:33.382 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.381+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3952693097 0 0) 0x7f38901380e0 con 0x7f389013f710 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.381+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38900abac0 con 0x7f389013f710 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.381+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3880001b40 con 0x7f389013f710 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.381+0000 7f3885ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.381+0000 7f3885ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.381+0000 7f3885ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:33.383 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:33.384 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1868805391 0 0) 0x7f38900abac0 con 0x7f389013f710 2024-03-25T18:21:33.387 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 >> v1:172.21.15.132:6790/0 conn(0x7f38901430c0 legacy=0x7f38901e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 >> v1:172.21.15.161:6789/0 conn(0x7f38900a98f0 legacy=0x7f38900ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 1 monclient: found mon.a 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3890137d90 con 0x7f389013f710 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f3885ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 5 monclient: authenticate success, global_id 4575 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 10 monclient: _renew_subs 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3890139240 con 0x7f389013f710 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 10 monclient: _renew_subs 2024-03-25T18:21:33.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:33.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3890136830 con 0x7f389013f710 2024-03-25T18:21:33.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.386+0000 7f389794b640 1 librados: init done 2024-03-25T18:21:33.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.387+0000 7f389794b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:33.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.387+0000 7f389794b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:33.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.387+0000 7f389794b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:33.389 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.387+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38900627b0 con 0x7f389013f710 2024-03-25T18:21:33.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.398+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38800011f0 con 0x7f389013f710 2024-03-25T18:21:33.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.398+0000 7f3885ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:33.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.398+0000 7f38877fe640 4 set_mon_vals no callback set 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.694+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3880004c30 con 0x7f389013f710 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.694+0000 7f3885ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.694+0000 7f3885ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.694+0000 7f3885ffb640 10 monclient: dump: 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:33.695 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.695+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f388001ac80 con 0x7f389013f710 2024-03-25T18:21:33.696 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.696+0000 7f38956e7640 10 monclient: get_auth_request con 0x7f387406e110 auth_method 0 2024-03-25T18:21:33.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.696+0000 7f3885ffb640 10 client.4575.objecter ms_handle_connect 0x7f387406e110 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f3880054970 con 0x7f389013f710 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 10 client.4575.objecter ms_dispatch 0x7f3890000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38800502b0 con 0x7f389013f710 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:33.707 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.706+0000 7f3885ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:33.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.967+0000 7f389794b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.967+0000 7f389794b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.967+0000 7f389794b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:21:33.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.967+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f389013f070 con 0x7f389013f710 2024-03-25T18:21:33.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.988+0000 7f3885ffb640 1 -- 172.21.15.132:0/3839018646 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f38800523b0 con 0x7f389013f710 2024-03-25T18:21:33.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.988+0000 7f3885ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.988+0000 7f3885ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:33.989 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:33.989 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 10 librados: watch_flush enter 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 10 librados: watch_flush exit 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 10 client.4575.objecter successfully canceled tick 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 >> v1:172.21.15.161:6800/411950978 conn(0x7f387406e110 legacy=0x7f38740705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 10 monclient: shutdown 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.990+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 >> v1:172.21.15.132:6789/0 conn(0x7f389013f710 legacy=0x7f3890138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.991+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 shutdown_connections 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.991+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 >> 172.21.15.132:0/3839018646 conn(0x7f38900b7ea0 msgr2=0x7f3890074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.991+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 shutdown_connections 2024-03-25T18:21:33.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.991+0000 7f389794b640 1 -- 172.21.15.132:0/3839018646 wait complete. 2024-03-25T18:21:33.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:33.991+0000 7f389794b640 1 librados: shutdown 2024-03-25T18:21:34.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:35.615 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:20.879 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.665 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:20.929 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.665 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:20.929 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.674 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:20.938 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.707 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:20.971 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.715 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:20.979 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.728 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:20.969 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.742 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:20.983 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:35.771 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:21.013 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:40.808 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:40.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.918+0000 7fd915684640 10 monclient: get_monmap_and_config 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.918+0000 7fd915684640 10 monclient: build_initial_monmap 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.918+0000 7fd915684640 10 monclient: monmap: 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:40.919586+0000 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:40.919586+0000 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:40.920 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.919+0000 7fd915684640 1 Processor -- start 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.919+0000 7fd915684640 1 -- start start 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.919+0000 7fd915684640 10 monclient: init 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient(hunting): picked mon.noname-a con 0x7fd9100a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient(hunting): picked mon.noname-b con 0x7fd91013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient(hunting): picked mon.noname-c con 0x7fd9101430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd910142520 con 0x7fd9100a98f0 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100b28a0 con 0x7fd9101430c0 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100b2a30 con 0x7fd91013f710 2024-03-25T18:21:40.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:40.923 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.920+0000 7fd915684640 10 monclient(hunting): authenticate will time out at 7741.183273s 2024-03-25T18:21:40.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.927+0000 7fd90e7fc640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fd91013f710 0x7fd9101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:43826/0 (socket says 172.21.15.132:43826) 2024-03-25T18:21:40.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.927+0000 7fd90e7fc640 1 -- 172.21.15.132:0/3277046900 learned_addr learned my addr 172.21.15.132:0/3277046900 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:40.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.932+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076172489 0 0) 0x7fd9100b2a30 con 0x7fd91013f710 2024-03-25T18:21:40.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.932+0000 7fd90dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.932+0000 7fd90dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.932+0000 7fd90dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.932+0000 7fd90dffb640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:21:40.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.932+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f8003520 con 0x7fd91013f710 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1431276757 0 0) 0x7fd8f8003520 con 0x7fd91013f710 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 >> v1:172.21.15.161:6789/0 conn(0x7fd9101430c0 legacy=0x7fd91014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 >> v1:172.21.15.132:6789/0 conn(0x7fd9100a98f0 legacy=0x7fd9100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 1 monclient: found mon.noname-b 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100b44c0 con 0x7fd91013f710 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:40.944 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd904001690 con 0x7fd91013f710 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd90dffb640 10 monclient: dump: 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:40.945 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:40.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd915684640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:21:40.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.942+0000 7fd915684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.952+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd904001860 con 0x7fd91013f710 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.952+0000 7fd90dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd90dffb640 1 -- 172.21.15.132:0/3277046900 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd904001cb0 con 0x7fd91013f710 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 4 set_mon_vals no callback set 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd90dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd90dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd90dffb640 10 monclient: dump: 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:40.954 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:40.955 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:40.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 10 monclient: shutdown 2024-03-25T18:21:40.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:40.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 1 -- 172.21.15.132:0/3277046900 >> v1:172.21.15.132:6790/0 conn(0x7fd91013f710 legacy=0x7fd9101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 1 -- 172.21.15.132:0/3277046900 shutdown_connections 2024-03-25T18:21:40.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 1 -- 172.21.15.132:0/3277046900 >> 172.21.15.132:0/3277046900 conn(0x7fd9100b7ea0 msgr2=0x7fd9100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:40.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.953+0000 7fd915684640 1 -- 172.21.15.132:0/3277046900 shutdown_connections 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.954+0000 7fd915684640 1 -- 172.21.15.132:0/3277046900 wait complete. 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.954+0000 7fd915684640 10 monclient: build_initial_monmap 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.954+0000 7fd915684640 10 monclient: monmap: 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:40.958 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.954+0000 7fd915684640 1 librados: starting msgr at 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.954+0000 7fd915684640 1 librados: starting objecter 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 Processor -- start 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 -- start start 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 librados: setting wanted keys 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 librados: calling monclient init 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient: init 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient(hunting): picked mon.noname-b con 0x7fd9100a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient(hunting): picked mon.noname-c con 0x7fd91013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient(hunting): picked mon.noname-a con 0x7fd9101430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9101d51f0 con 0x7fd9101430c0 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd910139040 con 0x7fd9100a98f0 2024-03-25T18:21:40.959 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9101dad70 con 0x7fd91013f710 2024-03-25T18:21:40.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:40.960 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.955+0000 7fd915684640 10 monclient(hunting): authenticate will time out at 7741.218638s 2024-03-25T18:21:40.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.974+0000 7fd90effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd9100a98f0 0x7fd910138930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:58886/0 (socket says 172.21.15.132:58886) 2024-03-25T18:21:40.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.974+0000 7fd90effd640 1 -- 172.21.15.132:0/2262894803 learned_addr learned my addr 172.21.15.132:0/2262894803 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:40.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.977+0000 7fd8f77fe640 10 client.?.objecter ms_handle_connect 0x7fd9100a98f0 2024-03-25T18:21:40.978 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.977+0000 7fd8f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.981+0000 7fd8f77fe640 10 client.?.objecter ms_handle_connect 0x7fd9101430c0 2024-03-25T18:21:40.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.981+0000 7fd8f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.984+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002280093 0 0) 0x7fd910139040 con 0x7fd9100a98f0 2024-03-25T18:21:40.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.984+0000 7fd8f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.984+0000 7fd8f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.984+0000 7fd8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.984+0000 7fd8f77fe640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:21:40.985 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.984+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc004eb0 con 0x7fd9100a98f0 2024-03-25T18:21:40.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.985+0000 7fd8f77fe640 10 client.?.objecter ms_handle_connect 0x7fd91013f710 2024-03-25T18:21:40.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.985+0000 7fd8f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.990 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.989+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071190016 0 0) 0x7fd9101dad70 con 0x7fd91013f710 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.990+0000 7fd8f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.990+0000 7fd8f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.990+0000 7fd8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.990+0000 7fd8f77fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:21:40.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.990+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd910139040 con 0x7fd91013f710 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.996+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2910826794 0 0) 0x7fd910139040 con 0x7fd91013f710 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.997+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9101dad70 con 0x7fd91013f710 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.997+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd904001b70 con 0x7fd91013f710 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.997+0000 7fd8f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.997+0000 7fd8f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.997+0000 7fd8f77fe640 10 monclient(hunting): dump: 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:40.998 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:40.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.997+0000 7fd8f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:41.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.998+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3692090259 0 0) 0x7fd8fc004eb0 con 0x7fd9100a98f0 2024-03-25T18:21:41.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.998+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd910139040 con 0x7fd9100a98f0 2024-03-25T18:21:41.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.998+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd900001960 con 0x7fd9100a98f0 2024-03-25T18:21:41.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.998+0000 7fd8f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.998+0000 7fd8f77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:41.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.998+0000 7fd8f77fe640 10 monclient(hunting): dump: 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.999+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512472385 0 0) 0x7fd9101d51f0 con 0x7fd9101430c0 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.999+0000 7fd8f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.999+0000 7fd8f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.999+0000 7fd8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.999+0000 7fd8f77fe640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:21:41.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:40.999+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8fc004eb0 con 0x7fd9101430c0 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.008+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2151237166 0 0) 0x7fd8fc004eb0 con 0x7fd9101430c0 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.008+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9101d51f0 con 0x7fd9101430c0 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.008+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd908001ad0 con 0x7fd9101430c0 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.008+0000 7fd8f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.008+0000 7fd8f77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.008+0000 7fd8f77fe640 10 monclient(hunting): dump: 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:41.010 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:41.011 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.011 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:41.011 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:41.011 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:41.011 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:41.011 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1572336600 0 0) 0x7fd9101dad70 con 0x7fd91013f710 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 >> v1:172.21.15.161:6789/0 conn(0x7fd9100a98f0 legacy=0x7fd910138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 >> v1:172.21.15.132:6789/0 conn(0x7fd9101430c0 legacy=0x7fd9101e1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 1 monclient: found mon.c 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9101db9a0 con 0x7fd91013f710 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:41.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd8f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 10 monclient: _renew_subs 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd910138150 con 0x7fd91013f710 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 10 monclient: _renew_subs 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9101e3610 con 0x7fd91013f710 2024-03-25T18:21:41.019 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.016+0000 7fd915684640 1 librados: init done 2024-03-25T18:21:41.020 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.017+0000 7fd915684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.020 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.017+0000 7fd915684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.020 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.017+0000 7fd915684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:41.020 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.017+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9100ac050 con 0x7fd91013f710 2024-03-25T18:21:41.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd904001cd0 con 0x7fd91013f710 2024-03-25T18:21:41.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd8f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:41.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd904001250 con 0x7fd91013f710 2024-03-25T18:21:41.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd8f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd8f77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:41.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd8f77fe640 10 monclient: dump: 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:41.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.342+0000 7fd90d7fa640 4 set_mon_vals no callback set 2024-03-25T18:21:41.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.343+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fd90401aa60 con 0x7fd91013f710 2024-03-25T18:21:41.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.343+0000 7fd90effd640 10 monclient: get_auth_request con 0x7fd8fc06e140 auth_method 0 2024-03-25T18:21:41.345 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.343+0000 7fd8f77fe640 10 client.4583.objecter ms_handle_connect 0x7fd8fc06e140 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fd904054600 con 0x7fd91013f710 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 10 client.4583.objecter ms_dispatch 0x7fd910000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd904050110 con 0x7fd91013f710 2024-03-25T18:21:41.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.351 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.349+0000 7fd8f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:41.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.571+0000 7fd915684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:41.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.571+0000 7fd915684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:41.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.571+0000 7fd915684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:41.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.571+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9101dab60 con 0x7fd91013f710 2024-03-25T18:21:41.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.604+0000 7fd8f77fe640 1 -- 172.21.15.132:0/2262894803 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fd904052040 con 0x7fd91013f710 2024-03-25T18:21:41.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.604+0000 7fd8f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:41.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.604+0000 7fd8f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:41.606 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:41.606 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 10 librados: watch_flush enter 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 10 librados: watch_flush exit 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 10 client.4583.objecter successfully canceled tick 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 >> v1:172.21.15.161:6800/411950978 conn(0x7fd8fc06e140 legacy=0x7fd8fc070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 10 monclient: shutdown 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 >> v1:172.21.15.132:6790/0 conn(0x7fd91013f710 legacy=0x7fd9101db290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 shutdown_connections 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 >> 172.21.15.132:0/2262894803 conn(0x7fd9100b7ea0 msgr2=0x7fd910074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 shutdown_connections 2024-03-25T18:21:41.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 -- 172.21.15.132:0/2262894803 wait complete. 2024-03-25T18:21:41.610 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:41.607+0000 7fd915684640 1 librados: shutdown 2024-03-25T18:21:41.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:48.426 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:48.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.544+0000 7f5b8d1e8640 10 monclient: get_monmap_and_config 2024-03-25T18:21:48.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.544+0000 7f5b8d1e8640 10 monclient: build_initial_monmap 2024-03-25T18:21:48.545 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.544+0000 7f5b8d1e8640 10 monclient: monmap: 2024-03-25T18:21:48.545 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:48.545 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:48.545434+0000 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:48.545434+0000 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:48.546 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.544+0000 7f5b8d1e8640 1 Processor -- start 2024-03-25T18:21:48.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 1 -- start start 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient: init 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b880a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b8813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b881430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b88142520 con 0x7f5b881430c0 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880b28a0 con 0x7f5b8813f710 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880b2a30 con 0x7f5b880a98f0 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:48.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.545+0000 7f5b8d1e8640 10 monclient(hunting): authenticate will time out at 7748.808548s 2024-03-25T18:21:48.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.549+0000 7f5b86d9d640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f5b880a98f0 0x7f5b880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:54920/0 (socket says 172.21.15.132:54920) 2024-03-25T18:21:48.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.549+0000 7f5b86d9d640 1 -- 172.21.15.132:0/444157351 learned_addr learned my addr 172.21.15.132:0/444157351 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:48.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.562+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754187678 0 0) 0x7f5b880b2a30 con 0x7f5b880a98f0 2024-03-25T18:21:48.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.562+0000 7f5b8659c640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.562+0000 7f5b8659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.562+0000 7f5b8659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.562+0000 7f5b8659c640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:21:48.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.562+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b74003520 con 0x7f5b880a98f0 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861626969 0 0) 0x7f5b88142520 con 0x7f5b881430c0 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b880b2a30 con 0x7f5b881430c0 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113314521 0 0) 0x7f5b880b28a0 con 0x7f5b8813f710 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:21:48.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.566+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b88142520 con 0x7f5b8813f710 2024-03-25T18:21:48.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2774686285 0 0) 0x7f5b74003520 con 0x7f5b880a98f0 2024-03-25T18:21:48.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 >> v1:172.21.15.161:6789/0 conn(0x7f5b8813f710 legacy=0x7f5b881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 >> v1:172.21.15.132:6789/0 conn(0x7f5b881430c0 legacy=0x7f5b8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: _finish_hunting 0 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 1 monclient: found mon.noname-b 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b880b44c0 con 0x7f5b880a98f0 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: _finish_auth 0 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: _check_auth_tickets 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5b78001690 con 0x7f5b880a98f0 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8659c640 10 monclient: dump: 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.573 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8d1e8640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:21:48.574 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.571+0000 7f5b8d1e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b780017f0 con 0x7f5b880a98f0 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8659c640 1 -- 172.21.15.132:0/444157351 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5b78001c00 con 0x7f5b880a98f0 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8d1e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8d1e8640 4 set_mon_vals no callback set 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.582+0000 7f5b8659c640 10 monclient: dump: 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:48.584 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 10 monclient: shutdown 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/444157351 >> v1:172.21.15.132:6790/0 conn(0x7f5b880a98f0 legacy=0x7f5b880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/444157351 shutdown_connections 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/444157351 >> 172.21.15.132:0/444157351 conn(0x7f5b880b7ea0 msgr2=0x7f5b880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/444157351 shutdown_connections 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.583+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/444157351 wait complete. 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.584+0000 7f5b8d1e8640 10 monclient: build_initial_monmap 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.584+0000 7f5b8d1e8640 10 monclient: monmap: 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:48.586 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.584+0000 7f5b8d1e8640 1 librados: starting msgr at 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.584+0000 7f5b8d1e8640 1 librados: starting objecter 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.584+0000 7f5b8d1e8640 1 Processor -- start 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.585+0000 7f5b8d1e8640 1 -- start start 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.585+0000 7f5b8d1e8640 1 librados: setting wanted keys 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.585+0000 7f5b8d1e8640 1 librados: calling monclient init 2024-03-25T18:21:48.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.585+0000 7f5b8d1e8640 10 monclient: init 2024-03-25T18:21:48.588 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient(hunting): picked mon.noname-b con 0x7f5b880a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient(hunting): picked mon.noname-a con 0x7f5b8813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient(hunting): picked mon.noname-c con 0x7f5b881430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b88138b20 con 0x7f5b8813f710 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b88137a80 con 0x7f5b880a98f0 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b88139880 con 0x7f5b881430c0 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:48.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.586+0000 7f5b8d1e8640 10 monclient(hunting): authenticate will time out at 7748.848984s 2024-03-25T18:21:48.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.591+0000 7f5b7e59c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f5b8813f710 0x7f5b88139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:44692/0 (socket says 172.21.15.132:44692) 2024-03-25T18:21:48.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.591+0000 7f5b7e59c640 1 -- 172.21.15.132:0/1185501474 learned_addr learned my addr 172.21.15.132:0/1185501474 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:48.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.592+0000 7f5b7ffff640 10 client.?.objecter ms_handle_connect 0x7f5b8813f710 2024-03-25T18:21:48.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.592+0000 7f5b7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:48.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.598+0000 7f5b7ffff640 10 client.?.objecter ms_handle_connect 0x7f5b881430c0 2024-03-25T18:21:48.599 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.598+0000 7f5b7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:48.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.601+0000 7f5b7ffff640 10 client.?.objecter ms_handle_connect 0x7f5b880a98f0 2024-03-25T18:21:48.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.601+0000 7f5b7ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657303006 0 0) 0x7f5b88139880 con 0x7f5b881430c0 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b68004eb0 con 0x7f5b881430c0 2024-03-25T18:21:48.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768897194 0 0) 0x7f5b88138b20 con 0x7f5b8813f710 2024-03-25T18:21:48.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:21:48.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.603+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b88139880 con 0x7f5b8813f710 2024-03-25T18:21:48.611 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.610+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2045119811 0 0) 0x7f5b68004eb0 con 0x7f5b881430c0 2024-03-25T18:21:48.611 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.610+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b88138b20 con 0x7f5b881430c0 2024-03-25T18:21:48.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.612+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255063463 0 0) 0x7f5b88137a80 con 0x7f5b880a98f0 2024-03-25T18:21:48.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.612+0000 7f5b7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:48.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.612+0000 7f5b7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:48.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.612+0000 7f5b7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:48.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.612+0000 7f5b7ffff640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:21:48.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.612+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b68004eb0 con 0x7f5b880a98f0 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.614+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5b80001cb0 con 0x7f5b881430c0 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.614+0000 7f5b7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.614+0000 7f5b7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.614+0000 7f5b7ffff640 10 monclient(hunting): dump: 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:48.615 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.614+0000 7f5b7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.622+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 488723814 0 0) 0x7f5b88139880 con 0x7f5b8813f710 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.622+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b88137a80 con 0x7f5b8813f710 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.622+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5b700019f0 con 0x7f5b8813f710 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.622+0000 7f5b7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.622+0000 7f5b7ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.622+0000 7f5b7ffff640 10 monclient(hunting): dump: 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.623 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:48.624 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:48.624 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:48.624 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:48.624 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2341757105 0 0) 0x7f5b88138b20 con 0x7f5b881430c0 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 >> v1:172.21.15.161:6789/0 conn(0x7f5b880a98f0 legacy=0x7f5b88137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 >> v1:172.21.15.132:6789/0 conn(0x7f5b8813f710 legacy=0x7f5b88139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 1 monclient: found mon.c 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88139a10 con 0x7f5b881430c0 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:48.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 10 monclient: _renew_subs 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b881dab00 con 0x7f5b881430c0 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 10 monclient: _renew_subs 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8813b4f0 con 0x7f5b881430c0 2024-03-25T18:21:48.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.630+0000 7f5b8d1e8640 1 librados: init done 2024-03-25T18:21:48.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.631+0000 7f5b8d1e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.631+0000 7f5b8d1e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.631+0000 7f5b8d1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:48.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.631+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b880b3e80 con 0x7f5b881430c0 2024-03-25T18:21:48.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.637+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b80003970 con 0x7f5b881430c0 2024-03-25T18:21:48.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.637+0000 7f5b7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:48.638 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.637+0000 7f5b85d9b640 4 set_mon_vals no callback set 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.657+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5b80003c80 con 0x7f5b881430c0 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.657+0000 7f5b7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.657+0000 7f5b7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.657+0000 7f5b7ffff640 10 monclient: dump: 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:48.658 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:48.659 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:48.659 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:48.659 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:48.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.658+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f5b80019c50 con 0x7f5b881430c0 2024-03-25T18:21:48.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.658+0000 7f5b86d9d640 10 monclient: get_auth_request con 0x7f5b6806df50 auth_method 0 2024-03-25T18:21:48.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.659+0000 7f5b7ffff640 10 client.4589.objecter ms_handle_connect 0x7f5b6806df50 2024-03-25T18:21:48.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.669+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f5b80053920 con 0x7f5b881430c0 2024-03-25T18:21:48.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.669+0000 7f5b7ffff640 10 client.4589.objecter ms_dispatch 0x7f5b88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:48.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.669+0000 7f5b7ffff640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:48.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.669+0000 7f5b7ffff640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:48.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.669+0000 7f5b7ffff640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T18:21:48.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.939+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b8004f260 con 0x7f5b881430c0 2024-03-25T18:21:48.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.939+0000 7f5b7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:48.939+0000 7f5b7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:49.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.189+0000 7f5b8d1e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:49.190 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.189+0000 7f5b8d1e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:49.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.189+0000 7f5b8d1e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:49.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.189+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b880627b0 con 0x7f5b881430c0 2024-03-25T18:21:49.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.216+0000 7f5b7ffff640 1 -- 172.21.15.132:0/1185501474 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f5b80051360 con 0x7f5b881430c0 2024-03-25T18:21:49.217 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.216+0000 7f5b7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:49.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.216+0000 7f5b7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:49.218 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:49.218 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 10 librados: watch_flush enter 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 10 librados: watch_flush exit 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 10 client.4589.objecter successfully canceled tick 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 >> v1:172.21.15.161:6800/411950978 conn(0x7f5b6806df50 legacy=0x7f5b68070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 10 monclient: shutdown 2024-03-25T18:21:49.220 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:49.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 >> v1:172.21.15.132:6790/0 conn(0x7f5b881430c0 legacy=0x7f5b881e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 shutdown_connections 2024-03-25T18:21:49.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.218+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 >> 172.21.15.132:0/1185501474 conn(0x7f5b880b7ea0 msgr2=0x7f5b88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:49.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.219+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 shutdown_connections 2024-03-25T18:21:49.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.219+0000 7f5b8d1e8640 1 -- 172.21.15.132:0/1185501474 wait complete. 2024-03-25T18:21:49.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:49.219+0000 7f5b8d1e8640 1 librados: shutdown 2024-03-25T18:21:49.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:55.518 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:40.782 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.564 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:40.829 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.600 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:08:40.864 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.621 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:40.862 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.626 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:40.868 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.633 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:40.897 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.640 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:40.904 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.661 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:08:40.903 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.666 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:08:40.930 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:56.038 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:56.154 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.153+0000 7f575e1fb640 10 monclient: get_monmap_and_config 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.153+0000 7f575e1fb640 10 monclient: build_initial_monmap 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.153+0000 7f575e1fb640 10 monclient: monmap: 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:21:56.154728+0000 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:21:56.154728+0000 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:21:56.155 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.154+0000 7f575e1fb640 1 Processor -- start 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.154+0000 7f575e1fb640 1 -- start start 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.154+0000 7f575e1fb640 10 monclient: init 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient(hunting): picked mon.noname-b con 0x7f57580a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient(hunting): picked mon.noname-a con 0x7f575813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient(hunting): picked mon.noname-c con 0x7f57581430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5758142520 con 0x7f575813f710 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580b28a0 con 0x7f57581430c0 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580b2a30 con 0x7f57580a98f0 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:56.157 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.155+0000 7f575e1fb640 10 monclient(hunting): authenticate will time out at 7756.418167s 2024-03-25T18:21:56.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.168+0000 7f57577fe640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f57580a98f0 0x7f57580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:60036/0 (socket says 172.21.15.132:60036) 2024-03-25T18:21:56.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.168+0000 7f57577fe640 1 -- 172.21.15.132:0/748823991 learned_addr learned my addr 172.21.15.132:0/748823991 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:56.177 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.176+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812334695 0 0) 0x7f57580b2a30 con 0x7f57580a98f0 2024-03-25T18:21:56.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.176+0000 7f5756ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.176+0000 7f5756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.176+0000 7f5756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.176+0000 7f5756ffd640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:21:56.178 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.176+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5744003520 con 0x7f57580a98f0 2024-03-25T18:21:56.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.182+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344396613 0 0) 0x7f57580b28a0 con 0x7f57581430c0 2024-03-25T18:21:56.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.182+0000 7f5756ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.182+0000 7f5756ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.182+0000 7f5756ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.182+0000 7f5756ffd640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:21:56.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.182+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57580b2a30 con 0x7f57581430c0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3312069541 0 0) 0x7f5744003520 con 0x7f57580a98f0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 >> v1:172.21.15.161:6789/0 conn(0x7f57581430c0 legacy=0x7f575814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 >> v1:172.21.15.132:6789/0 conn(0x7f575813f710 legacy=0x7f57581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 1 monclient: found mon.noname-b 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580b44c0 con 0x7f57580a98f0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5748001690 con 0x7f57580a98f0 2024-03-25T18:21:56.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f5756ffd640 10 monclient: dump: 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f575e1fb640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:21:56.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.183+0000 7f575e1fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:56.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57480017f0 con 0x7f57580a98f0 2024-03-25T18:21:56.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f5756ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:56.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f5756ffd640 1 -- 172.21.15.132:0/748823991 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5748001c00 con 0x7f57580a98f0 2024-03-25T18:21:56.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f5756ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f5756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f5756ffd640 10 monclient: dump: 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f575e1fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:56.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.201+0000 7f575e1fb640 4 set_mon_vals no callback set 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 10 monclient: shutdown 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 1 -- 172.21.15.132:0/748823991 >> v1:172.21.15.132:6790/0 conn(0x7f57580a98f0 legacy=0x7f57580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 1 -- 172.21.15.132:0/748823991 shutdown_connections 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 1 -- 172.21.15.132:0/748823991 >> 172.21.15.132:0/748823991 conn(0x7f57580b7ea0 msgr2=0x7f57580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 1 -- 172.21.15.132:0/748823991 shutdown_connections 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.202+0000 7f575e1fb640 1 -- 172.21.15.132:0/748823991 wait complete. 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 10 monclient: build_initial_monmap 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 10 monclient: monmap: 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:21:56.206 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 1 librados: starting msgr at 2024-03-25T18:21:56.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 1 librados: starting objecter 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 1 Processor -- start 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 1 -- start start 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 1 librados: setting wanted keys 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 1 librados: calling monclient init 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.203+0000 7f575e1fb640 10 monclient: init 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient(hunting): picked mon.noname-c con 0x7f57580a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient(hunting): picked mon.noname-b con 0x7f575813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient(hunting): picked mon.noname-a con 0x7f57581430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:21:56.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57581db7d0 con 0x7f57581430c0 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580b1420 con 0x7f575813f710 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580b15b0 con 0x7f57580a98f0 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f575e1fb640 10 monclient(hunting): authenticate will time out at 7756.467037s 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f5757fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f57581430c0 0x7f57581d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:59842/0 (socket says 172.21.15.132:59842) 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f5757fff640 1 -- 172.21.15.132:0/3072775175 learned_addr learned my addr 172.21.15.132:0/3072775175 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f5754ff9640 10 client.?.objecter ms_handle_connect 0x7f57581430c0 2024-03-25T18:21:56.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.204+0000 7f5754ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:56.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.217+0000 7f5754ff9640 10 client.?.objecter ms_handle_connect 0x7f575813f710 2024-03-25T18:21:56.218 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.217+0000 7f5754ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:56.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.230+0000 7f5754ff9640 10 client.?.objecter ms_handle_connect 0x7f57580a98f0 2024-03-25T18:21:56.232 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.230+0000 7f5754ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:56.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.247+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2045251622 0 0) 0x7f57580b15b0 con 0x7f57580a98f0 2024-03-25T18:21:56.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.247+0000 7f5754ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.247+0000 7f5754ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.247+0000 7f5754ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.248 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.247+0000 7f5754ff9640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:21:56.249 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.247+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5738004eb0 con 0x7f57580a98f0 2024-03-25T18:21:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.320+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1386514314 0 0) 0x7f57581db7d0 con 0x7f57581430c0 2024-03-25T18:21:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.320+0000 7f5754ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.320+0000 7f5754ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.320+0000 7f5754ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.320+0000 7f5754ff9640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:21:56.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.321+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57580b15b0 con 0x7f57581430c0 2024-03-25T18:21:56.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3745470593 0 0) 0x7f5738004eb0 con 0x7f57580a98f0 2024-03-25T18:21:56.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57581db7d0 con 0x7f57580a98f0 2024-03-25T18:21:56.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5748001b00 con 0x7f57580a98f0 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.324+0000 7f5754ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:56.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.325+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501592815 0 0) 0x7f57580b1420 con 0x7f575813f710 2024-03-25T18:21:56.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.325+0000 7f5754ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:56.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.325+0000 7f5754ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:56.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.325+0000 7f5754ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:56.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.325+0000 7f5754ff9640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:21:56.327 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.325+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5738004eb0 con 0x7f575813f710 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.340+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4160965483 0 0) 0x7f57580b15b0 con 0x7f57581430c0 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.340+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57580b1420 con 0x7f57581430c0 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.340+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5750001c50 con 0x7f57581430c0 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.340+0000 7f5754ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.340+0000 7f5754ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.340+0000 7f5754ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:56.341 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:56.342 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.348+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4154796622 0 0) 0x7f5738004eb0 con 0x7f575813f710 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.348+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57580b15b0 con 0x7f575813f710 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.348+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5740001870 con 0x7f575813f710 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.348+0000 7f5754ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.348+0000 7f5754ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.348+0000 7f5754ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:56.350 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:56.351 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:56.351 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2410369631 0 0) 0x7f57581db7d0 con 0x7f57580a98f0 2024-03-25T18:21:56.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 >> v1:172.21.15.161:6789/0 conn(0x7f575813f710 legacy=0x7f57580b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 >> v1:172.21.15.132:6789/0 conn(0x7f57581430c0 legacy=0x7f57581d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 1 monclient: found mon.c 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580b1740 con 0x7f57580a98f0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 10 monclient: _finish_auth 0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f5754ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 10 monclient: _renew_subs 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57581db1a0 con 0x7f57580a98f0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 10 monclient: _renew_subs 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57580b08f0 con 0x7f57580a98f0 2024-03-25T18:21:56.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.358+0000 7f575e1fb640 1 librados: init done 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.359+0000 7f575e1fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.359+0000 7f575e1fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.359+0000 7f575e1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:56.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.359+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57580627b0 con 0x7f57580a98f0 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5748001220 con 0x7f57580a98f0 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f5754ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f5748004c30 con 0x7f57580a98f0 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f5754ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f5754ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f5754ff9640 10 monclient: dump: 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:56.651 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.649+0000 7f57567fc640 4 set_mon_vals no callback set 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.650+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f574801ac00 con 0x7f57580a98f0 2024-03-25T18:21:56.652 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.651+0000 7f574ffff640 10 monclient: get_auth_request con 0x7f573806e100 auth_method 0 2024-03-25T18:21:56.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.651+0000 7f5754ff9640 10 client.4595.objecter ms_handle_connect 0x7f573806e100 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f5748054840 con 0x7f57580a98f0 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 10 client.4595.objecter ms_dispatch 0x7f5758000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5748050350 con 0x7f57580a98f0 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:56.660 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.658+0000 7f5754ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:56.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.910+0000 7f575e1fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.910+0000 7f575e1fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.910+0000 7f575e1fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:21:56.912 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.910+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57580abae0 con 0x7f57580a98f0 2024-03-25T18:21:56.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.940+0000 7f5754ff9640 1 -- 172.21.15.132:0/3072775175 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f5748052280 con 0x7f57580a98f0 2024-03-25T18:21:56.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.940+0000 7f5754ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.940+0000 7f5754ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:56.942 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:21:56.942 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:56.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 10 librados: watch_flush enter 2024-03-25T18:21:56.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 10 librados: watch_flush exit 2024-03-25T18:21:56.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T18:21:56.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 10 client.4595.objecter successfully canceled tick 2024-03-25T18:21:56.945 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 >> v1:172.21.15.161:6800/411950978 conn(0x7f573806e100 legacy=0x7f57380705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 10 monclient: shutdown 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.943+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 >> v1:172.21.15.132:6790/0 conn(0x7f57580a98f0 legacy=0x7f57581daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.944+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 shutdown_connections 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.944+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 >> 172.21.15.132:0/3072775175 conn(0x7f57580b7ea0 msgr2=0x7f575813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.944+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 shutdown_connections 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.944+0000 7f575e1fb640 1 -- 172.21.15.132:0/3072775175 wait complete. 2024-03-25T18:21:56.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:21:56.945+0000 7f575e1fb640 1 librados: shutdown 2024-03-25T18:21:56.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:03.764 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:03.875 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.874+0000 7f8a3c04a640 10 monclient: get_monmap_and_config 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.874+0000 7f8a3c04a640 10 monclient: build_initial_monmap 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.874+0000 7f8a3c04a640 10 monclient: monmap: 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:03.875942+0000 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:03.875942+0000 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:03.876 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:03.877 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.875+0000 7f8a3c04a640 1 Processor -- start 2024-03-25T18:22:03.877 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.875+0000 7f8a3c04a640 1 -- start start 2024-03-25T18:22:03.877 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.875+0000 7f8a3c04a640 10 monclient: init 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient(hunting): picked mon.noname-a con 0x7f8a340b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient(hunting): picked mon.noname-b con 0x7f8a340b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient(hunting): picked mon.noname-c con 0x7f8a3414be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a340af940 con 0x7f8a340b4d60 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a340b0a70 con 0x7f8a3414be90 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a340b0d50 con 0x7f8a340b1730 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:03.878 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.876+0000 7f8a3c04a640 10 monclient(hunting): authenticate will time out at 7764.139198s 2024-03-25T18:22:03.881 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.880+0000 7f8a395e5640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f8a340b1730 0x7f8a340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:54848/0 (socket says 172.21.15.132:54848) 2024-03-25T18:22:03.881 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.880+0000 7f8a395e5640 1 -- 172.21.15.132:0/809986721 learned_addr learned my addr 172.21.15.132:0/809986721 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:03.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.888+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503578179 0 0) 0x7f8a340b0d50 con 0x7f8a340b1730 2024-03-25T18:22:03.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.888+0000 7f8a38de4640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.889 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.888+0000 7f8a38de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.888+0000 7f8a38de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.888+0000 7f8a38de4640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:22:03.890 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.888+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a20003520 con 0x7f8a340b1730 2024-03-25T18:22:03.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.892+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591144055 0 0) 0x7f8a340b0a70 con 0x7f8a3414be90 2024-03-25T18:22:03.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.892+0000 7f8a38de4640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.892+0000 7f8a38de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.892+0000 7f8a38de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.892+0000 7f8a38de4640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:22:03.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.892+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a340b0d50 con 0x7f8a3414be90 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722384864 0 0) 0x7f8a20003520 con 0x7f8a340b1730 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 >> v1:172.21.15.161:6789/0 conn(0x7f8a3414be90 legacy=0x7f8a3414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 >> v1:172.21.15.132:6789/0 conn(0x7f8a340b4d60 legacy=0x7f8a3413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: _finish_hunting 0 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 1 monclient: found mon.noname-b 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a340b1030 con 0x7f8a340b1730 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: _finish_auth 0 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: _check_auth_tickets 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a24001690 con 0x7f8a340b1730 2024-03-25T18:22:03.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a3c04a640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a3c04a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a38de4640 10 monclient: dump: 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:03.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.904+0000 7f8a3c04a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:03.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a240017f0 con 0x7f8a340b1730 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a38de4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a38de4640 1 -- 172.21.15.132:0/809986721 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a24001c00 con 0x7f8a340b1730 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a38de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a38de4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a38de4640 10 monclient: dump: 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:03.917 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:03.918 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:03.918 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:03.918 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:03.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a3c04a640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:03.918 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.915+0000 7f8a3c04a640 4 set_mon_vals no callback set 2024-03-25T18:22:03.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 10 monclient: shutdown 2024-03-25T18:22:03.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:03.919 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 1 -- 172.21.15.132:0/809986721 >> v1:172.21.15.132:6790/0 conn(0x7f8a340b1730 legacy=0x7f8a340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 1 -- 172.21.15.132:0/809986721 shutdown_connections 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 1 -- 172.21.15.132:0/809986721 >> 172.21.15.132:0/809986721 conn(0x7f8a340617a0 msgr2=0x7f8a34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 1 -- 172.21.15.132:0/809986721 shutdown_connections 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.916+0000 7f8a3c04a640 1 -- 172.21.15.132:0/809986721 wait complete. 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.917+0000 7f8a3c04a640 10 monclient: build_initial_monmap 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.917+0000 7f8a3c04a640 10 monclient: monmap: 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:03.920 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 1 librados: starting msgr at 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 1 librados: starting objecter 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 1 Processor -- start 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 1 -- start start 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 1 librados: setting wanted keys 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 1 librados: calling monclient init 2024-03-25T18:22:03.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.918+0000 7f8a3c04a640 10 monclient: init 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient(hunting): picked mon.noname-a con 0x7f8a340b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient(hunting): picked mon.noname-c con 0x7f8a340b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3414be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a341d7690 con 0x7f8a340b1730 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a341d7970 con 0x7f8a3414be90 2024-03-25T18:22:03.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a341398e0 con 0x7f8a340b4d60 2024-03-25T18:22:03.923 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:03.923 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.919+0000 7f8a3c04a640 10 monclient(hunting): authenticate will time out at 7764.181972s 2024-03-25T18:22:03.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.932+0000 7f8a395e5640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f8a340b4d60 0x7f8a341d7e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:54872/0 (socket says 172.21.15.132:54872) 2024-03-25T18:22:03.933 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.932+0000 7f8a395e5640 1 -- 172.21.15.132:0/3782301870 learned_addr learned my addr 172.21.15.132:0/3782301870 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:03.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.933+0000 7f8a2a7fc640 10 client.?.objecter ms_handle_connect 0x7f8a340b4d60 2024-03-25T18:22:03.934 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.933+0000 7f8a2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.936+0000 7f8a2a7fc640 10 client.?.objecter ms_handle_connect 0x7f8a3414be90 2024-03-25T18:22:03.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.936+0000 7f8a2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.937+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808577688 0 0) 0x7f8a341398e0 con 0x7f8a340b4d60 2024-03-25T18:22:03.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.937+0000 7f8a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.937+0000 7f8a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.937+0000 7f8a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.937+0000 7f8a2a7fc640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:22:03.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.937+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18004eb0 con 0x7f8a340b4d60 2024-03-25T18:22:03.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.938+0000 7f8a2a7fc640 10 client.?.objecter ms_handle_connect 0x7f8a340b1730 2024-03-25T18:22:03.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.938+0000 7f8a2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.945+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1795087865 0 0) 0x7f8a18004eb0 con 0x7f8a340b4d60 2024-03-25T18:22:03.946 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.945+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a341398e0 con 0x7f8a340b4d60 2024-03-25T18:22:03.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.947+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a240047e0 con 0x7f8a340b4d60 2024-03-25T18:22:03.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.947+0000 7f8a2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.947+0000 7f8a2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:03.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.947+0000 7f8a2a7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:03.948 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:03.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.947+0000 7f8a2a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:03.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.953+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028006519 0 0) 0x7f8a341d7970 con 0x7f8a3414be90 2024-03-25T18:22:03.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.953+0000 7f8a2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.953+0000 7f8a2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.953+0000 7f8a2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.953+0000 7f8a2a7fc640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:22:03.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.953+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18004eb0 con 0x7f8a3414be90 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3393110512 0 0) 0x7f8a341398e0 con 0x7f8a340b4d60 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 >> v1:172.21.15.161:6789/0 conn(0x7f8a3414be90 legacy=0x7f8a3413b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 >> v1:172.21.15.132:6789/0 conn(0x7f8a340b1730 legacy=0x7f8a341391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 1 monclient: found mon.c 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a341d8510 con 0x7f8a340b4d60 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 5 monclient: authenticate success, global_id 4601 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 10 monclient: _renew_subs 2024-03-25T18:22:03.956 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a341433d0 con 0x7f8a340b4d60 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 10 client.4601.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 10 monclient: _renew_subs 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3413bcf0 con 0x7f8a340b4d60 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.954+0000 7f8a3c04a640 1 librados: init done 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.955+0000 7f8a3c04a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.955+0000 7f8a3c04a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.957 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.955+0000 7f8a3c04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:03.958 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.955+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a340627b0 con 0x7f8a340b4d60 2024-03-25T18:22:03.961 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.959+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a24004bf0 con 0x7f8a340b4d60 2024-03-25T18:22:03.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.960+0000 7f8a2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:03.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.960+0000 7f8a2bfff640 4 set_mon_vals no callback set 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.962+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f8a24004f00 con 0x7f8a340b4d60 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.962+0000 7f8a2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.962+0000 7f8a2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:03.962+0000 7f8a2a7fc640 10 monclient: dump: 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.963 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:03.964 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.964 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:03.964 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:03.964 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:03.964 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:03.964 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:04.265 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.264+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f8a2401aed0 con 0x7f8a340b4d60 2024-03-25T18:22:04.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.265+0000 7f8a39de6640 10 monclient: get_auth_request con 0x7f8a1806dce0 auth_method 0 2024-03-25T18:22:04.266 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.265+0000 7f8a2a7fc640 10 client.4601.objecter ms_handle_connect 0x7f8a1806dce0 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f8a24054af0 con 0x7f8a340b4d60 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 10 client.4601.objecter ms_dispatch 0x7f8a34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 3 client.4601.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 3 client.4601.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 20 client.4601.objecter dump_active .. 0 homeless 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a24050430 con 0x7f8a340b4d60 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:04.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.270+0000 7f8a2a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:04.491 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.490+0000 7f8a3c04a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:04.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.490+0000 7f8a3c04a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:04.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.490+0000 7f8a3c04a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:04.492 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.490+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a34069f40 con 0x7f8a340b4d60 2024-03-25T18:22:04.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.520+0000 7f8a2a7fc640 1 -- 172.21.15.132:0/3782301870 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f8a24052530 con 0x7f8a340b4d60 2024-03-25T18:22:04.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.520+0000 7f8a2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:04.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.520+0000 7f8a2a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:04.522 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:04.522 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 10 librados: watch_flush enter 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 10 librados: watch_flush exit 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 20 client.4601.objecter shutdown clearing up homeless session... 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 10 client.4601.objecter successfully canceled tick 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 >> v1:172.21.15.161:6800/411950978 conn(0x7f8a1806dce0 legacy=0x7f8a180701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 10 monclient: shutdown 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 >> v1:172.21.15.132:6790/0 conn(0x7f8a340b4d60 legacy=0x7f8a341d7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 shutdown_connections 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 >> 172.21.15.132:0/3782301870 conn(0x7f8a340617a0 msgr2=0x7f8a34137280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 shutdown_connections 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 -- 172.21.15.132:0/3782301870 wait complete. 2024-03-25T18:22:04.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:04.523+0000 7f8a3c04a640 1 librados: shutdown 2024-03-25T18:22:04.537 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:11.344 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:11.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.462+0000 7f0f26e6e640 10 monclient: get_monmap_and_config 2024-03-25T18:22:11.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.462+0000 7f0f26e6e640 10 monclient: build_initial_monmap 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.462+0000 7f0f26e6e640 10 monclient: monmap: 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:11.463503+0000 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:11.463503+0000 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:11.464 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:11.465 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.463+0000 7f0f26e6e640 1 Processor -- start 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.463+0000 7f0f26e6e640 1 -- start start 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.463+0000 7f0f26e6e640 10 monclient: init 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient(hunting): picked mon.noname-b con 0x7f0f200a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient(hunting): picked mon.noname-a con 0x7f0f2013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient(hunting): picked mon.noname-c con 0x7f0f201430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f20142520 con 0x7f0f2013f710 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200b28a0 con 0x7f0f201430c0 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200b2a30 con 0x7f0f200a98f0 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:11.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.464+0000 7f0f26e6e640 10 monclient(hunting): authenticate will time out at 7771.727023s 2024-03-25T18:22:11.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.469+0000 7f0f1ffff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f0f2013f710 0x7f0f201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37094/0 (socket says 172.21.15.132:37094) 2024-03-25T18:22:11.470 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.469+0000 7f0f1ffff640 1 -- 172.21.15.132:0/2333064957 learned_addr learned my addr 172.21.15.132:0/2333064957 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:11.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.479+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91216795 0 0) 0x7f0f20142520 con 0x7f0f2013f710 2024-03-25T18:22:11.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.479+0000 7f0f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.479+0000 7f0f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.479+0000 7f0f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.479+0000 7f0f1f7fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:22:11.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.479+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f10003520 con 0x7f0f2013f710 2024-03-25T18:22:11.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.492+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856139882 0 0) 0x7f0f200b28a0 con 0x7f0f201430c0 2024-03-25T18:22:11.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.492+0000 7f0f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.492+0000 7f0f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.492+0000 7f0f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.492+0000 7f0f1f7fe640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:22:11.493 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.492+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f20142520 con 0x7f0f201430c0 2024-03-25T18:22:11.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.494+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881007260 0 0) 0x7f0f200b2a30 con 0x7f0f200a98f0 2024-03-25T18:22:11.495 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.494+0000 7f0f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.494+0000 7f0f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.494+0000 7f0f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.494+0000 7f0f1f7fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:22:11.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.494+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f200b28a0 con 0x7f0f200a98f0 2024-03-25T18:22:11.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4033652542 0 0) 0x7f0f10003520 con 0x7f0f2013f710 2024-03-25T18:22:11.504 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 >> v1:172.21.15.132:6790/0 conn(0x7f0f200a98f0 legacy=0x7f0f200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 >> v1:172.21.15.161:6789/0 conn(0x7f0f201430c0 legacy=0x7f0f2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f200b44c0 con 0x7f0f2013f710 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0f0c001690 con 0x7f0f2013f710 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f1f7fe640 10 monclient: dump: 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:11.505 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f26e6e640 5 monclient: authenticate success, global_id 4593 2024-03-25T18:22:11.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.503+0000 7f0f26e6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f0c0017f0 con 0x7f0f2013f710 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f1f7fe640 1 -- 172.21.15.132:0/2333064957 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0f0c001c00 con 0x7f0f2013f710 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f26e6e640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f26e6e640 4 set_mon_vals no callback set 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:11.512 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.511+0000 7f0f1f7fe640 10 monclient: dump: 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:11.513 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 10 monclient: shutdown 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2333064957 >> v1:172.21.15.132:6789/0 conn(0x7f0f2013f710 legacy=0x7f0f201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2333064957 shutdown_connections 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2333064957 >> 172.21.15.132:0/2333064957 conn(0x7f0f200b7ea0 msgr2=0x7f0f200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2333064957 shutdown_connections 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.512+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2333064957 wait complete. 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 10 monclient: build_initial_monmap 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 10 monclient: monmap: 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:11.515 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 1 librados: starting msgr at 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 1 librados: starting objecter 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 1 Processor -- start 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 1 -- start start 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 1 librados: setting wanted keys 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 1 librados: calling monclient init 2024-03-25T18:22:11.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.513+0000 7f0f26e6e640 10 monclient: init 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient(hunting): picked mon.noname-c con 0x7f0f200a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient(hunting): picked mon.noname-a con 0x7f0f2013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient(hunting): picked mon.noname-b con 0x7f0f201430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f201d82d0 con 0x7f0f2013f710 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f200abb90 con 0x7f0f201430c0 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f201d76a0 con 0x7f0f200a98f0 2024-03-25T18:22:11.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:11.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.514+0000 7f0f26e6e640 10 monclient(hunting): authenticate will time out at 7771.777517s 2024-03-25T18:22:11.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.522+0000 7f0f1ffff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f0f2013f710 0x7f0f201d7bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37120/0 (socket says 172.21.15.132:37120) 2024-03-25T18:22:11.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.522+0000 7f0f1ffff640 1 -- 172.21.15.132:0/2112247232 learned_addr learned my addr 172.21.15.132:0/2112247232 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:11.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.524+0000 7f0f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f0f2013f710 2024-03-25T18:22:11.525 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.524+0000 7f0f1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:11.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.525+0000 7f0f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f0f201430c0 2024-03-25T18:22:11.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.525+0000 7f0f1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:11.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.525+0000 7f0f1d7fa640 10 client.?.objecter ms_handle_connect 0x7f0f200a98f0 2024-03-25T18:22:11.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.525+0000 7f0f1d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:11.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.528+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698018435 0 0) 0x7f0f201d82d0 con 0x7f0f2013f710 2024-03-25T18:22:11.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.529+0000 7f0f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.529+0000 7f0f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.529+0000 7f0f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.529+0000 7f0f1d7fa640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:22:11.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.529+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f04004eb0 con 0x7f0f2013f710 2024-03-25T18:22:11.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.533+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680829027 0 0) 0x7f0f200abb90 con 0x7f0f201430c0 2024-03-25T18:22:11.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.533+0000 7f0f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.533+0000 7f0f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.533+0000 7f0f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.533+0000 7f0f1d7fa640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:22:11.534 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.533+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f201d82d0 con 0x7f0f201430c0 2024-03-25T18:22:11.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.534+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536599273 0 0) 0x7f0f201d76a0 con 0x7f0f200a98f0 2024-03-25T18:22:11.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.534+0000 7f0f1d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:11.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.534+0000 7f0f1d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:11.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.534+0000 7f0f1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:11.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.534+0000 7f0f1d7fa640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:22:11.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.534+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f200abb90 con 0x7f0f200a98f0 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.540+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135319758 0 0) 0x7f0f04004eb0 con 0x7f0f2013f710 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.540+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f201d76a0 con 0x7f0f2013f710 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.540+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0f0c001bc0 con 0x7f0f2013f710 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.540+0000 7f0f1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.540+0000 7f0f1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.540+0000 7f0f1d7fa640 10 monclient(hunting): dump: 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:11.542 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:11.543 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:11.543 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:11.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.541+0000 7f0f1d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:11.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3237490133 0 0) 0x7f0f201d76a0 con 0x7f0f2013f710 2024-03-25T18:22:11.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 >> v1:172.21.15.132:6790/0 conn(0x7f0f200a98f0 legacy=0x7f0f201385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 >> v1:172.21.15.161:6789/0 conn(0x7f0f201430c0 legacy=0x7f0f20134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.550 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 1 monclient: found mon.a 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f20134720 con 0x7f0f2013f710 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f26e6e640 5 monclient: authenticate success, global_id 4596 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f26e6e640 10 monclient: _renew_subs 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f26e6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.548+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f20134970 con 0x7f0f2013f710 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.549+0000 7f0f26e6e640 10 client.4596.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:11.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.549+0000 7f0f26e6e640 10 monclient: _renew_subs 2024-03-25T18:22:11.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.549+0000 7f0f26e6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:22:11.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.549+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f20137a10 con 0x7f0f2013f710 2024-03-25T18:22:11.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.549+0000 7f0f26e6e640 1 librados: init done 2024-03-25T18:22:11.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.550+0000 7f0f26e6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.550+0000 7f0f26e6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.553 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.550+0000 7f0f26e6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:22:11.553 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.550+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f201d7490 con 0x7f0f2013f710 2024-03-25T18:22:11.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f0c0011f0 con 0x7f0f2013f710 2024-03-25T18:22:11.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:11.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0f0c004cb0 con 0x7f0f2013f710 2024-03-25T18:22:11.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:11.861 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 10 monclient: dump: 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:11.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1effd640 4 set_mon_vals no callback set 2024-03-25T18:22:11.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.860+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f0f0c01ad00 con 0x7f0f2013f710 2024-03-25T18:22:11.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.861+0000 7f0f24c0a640 10 monclient: get_auth_request con 0x7f0f0406dd50 auth_method 0 2024-03-25T18:22:11.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.862+0000 7f0f1d7fa640 10 client.4596.objecter ms_handle_connect 0x7f0f0406dd50 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f0f0c0549d0 con 0x7f0f2013f710 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 10 client.4596.objecter ms_dispatch 0x7f0f20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 3 client.4596.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 3 client.4596.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 20 client.4596.objecter dump_active .. 0 homeless 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f0c050310 con 0x7f0f2013f710 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:11.866+0000 7f0f1d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:12.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.120+0000 7f0f26e6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:12.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.120+0000 7f0f26e6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:12.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.120+0000 7f0f26e6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:22:12.123 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.120+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f200627b0 con 0x7f0f2013f710 2024-03-25T18:22:12.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.138+0000 7f0f1d7fa640 1 -- 172.21.15.132:0/2112247232 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f0f0c052410 con 0x7f0f2013f710 2024-03-25T18:22:12.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.138+0000 7f0f1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:12.140 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.138+0000 7f0f1d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:12.140 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:12.140 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:12.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.140+0000 7f0f26e6e640 10 librados: watch_flush enter 2024-03-25T18:22:12.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.140+0000 7f0f26e6e640 10 librados: watch_flush exit 2024-03-25T18:22:12.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.140+0000 7f0f26e6e640 20 client.4596.objecter shutdown clearing up homeless session... 2024-03-25T18:22:12.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.140+0000 7f0f26e6e640 10 client.4596.objecter successfully canceled tick 2024-03-25T18:22:12.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.140+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 >> v1:172.21.15.161:6800/411950978 conn(0x7f0f0406dd50 legacy=0x7f0f04070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.142 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 10 monclient: shutdown 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 >> v1:172.21.15.132:6789/0 conn(0x7f0f2013f710 legacy=0x7f0f201d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 shutdown_connections 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 >> 172.21.15.132:0/2112247232 conn(0x7f0f200b7ea0 msgr2=0x7f0f20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 shutdown_connections 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 1 -- 172.21.15.132:0/2112247232 wait complete. 2024-03-25T18:22:12.143 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:12.141+0000 7f0f26e6e640 1 librados: shutdown 2024-03-25T18:22:12.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:15.747 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:01.011 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.751 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:01.015 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.754 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:01.018 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.765 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:01.029 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.786 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:01.050 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.787 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:01.051 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.819 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:01.061 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.825 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:01.067 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:15.855 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:01.097 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:18.960 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.074+0000 7f48acdf4640 10 monclient: get_monmap_and_config 2024-03-25T18:22:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.074+0000 7f48acdf4640 10 monclient: build_initial_monmap 2024-03-25T18:22:19.115 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.074+0000 7f48acdf4640 10 monclient: monmap: 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:19.075433+0000 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:19.075433+0000 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.075+0000 7f48acdf4640 1 Processor -- start 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.075+0000 7f48acdf4640 1 -- start start 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.075+0000 7f48acdf4640 10 monclient: init 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:19.116 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient(hunting): picked mon.noname-b con 0x7f48a806e200 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient(hunting): picked mon.noname-c con 0x7f48a8070b50 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient(hunting): picked mon.noname-a con 0x7f48a8152ec0 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a806e990 con 0x7f48a8152ec0 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80701e0 con 0x7f48a8070b50 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a806f6b0 con 0x7f48a806e200 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.076+0000 7f48acdf4640 10 monclient(hunting): authenticate will time out at 7779.339105s 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.079+0000 7f48a5d9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f48a8070b50 0x7f48a80af480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:59880/0 (socket says 172.21.15.132:59880) 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.079+0000 7f48a5d9b640 1 -- 172.21.15.132:0/317171496 learned_addr learned my addr 172.21.15.132:0/317171496 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.085+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380582286 0 0) 0x7f48a80701e0 con 0x7f48a8070b50 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.085+0000 7f48a559a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.085+0000 7f48a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.085+0000 7f48a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.085+0000 7f48a559a640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:22:19.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.085+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4890003520 con 0x7f48a8070b50 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.087+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236634980 0 0) 0x7f48a806f6b0 con 0x7f48a806e200 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a80701e0 con 0x7f48a806e200 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418085159 0 0) 0x7f4890003520 con 0x7f48a8070b50 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 >> v1:172.21.15.132:6790/0 conn(0x7f48a806e200 legacy=0x7f48a81527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 >> v1:172.21.15.132:6789/0 conn(0x7f48a8152ec0 legacy=0x7f48a815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: _finish_hunting 0 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 monclient: found mon.noname-c 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:22:19.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a8070450 con 0x7f48a8070b50 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: _finish_auth 0 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: _check_auth_tickets 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f489c001690 con 0x7f48a8070b50 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48a559a640 10 monclient: dump: 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:19.119 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:19.120 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:19.120 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:19.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48acdf4640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:22:19.120 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.113+0000 7f48acdf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:19.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f489c0018b0 con 0x7f48a8070b50 2024-03-25T18:22:19.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48a559a640 1 -- 172.21.15.132:0/317171496 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f489c001c40 con 0x7f48a8070b50 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48acdf4640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48acdf4640 4 set_mon_vals no callback set 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.130+0000 7f48a559a640 10 monclient: dump: 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:19.132 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:19.133 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.131+0000 7f48acdf4640 10 monclient: shutdown 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.131+0000 7f48acdf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.131+0000 7f48acdf4640 1 -- 172.21.15.132:0/317171496 >> v1:172.21.15.161:6789/0 conn(0x7f48a8070b50 legacy=0x7f48a80af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.131+0000 7f48acdf4640 1 -- 172.21.15.132:0/317171496 shutdown_connections 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.131+0000 7f48acdf4640 1 -- 172.21.15.132:0/317171496 >> 172.21.15.132:0/317171496 conn(0x7f48a806b630 msgr2=0x7f48a80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.131+0000 7f48acdf4640 1 -- 172.21.15.132:0/317171496 shutdown_connections 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.132+0000 7f48acdf4640 1 -- 172.21.15.132:0/317171496 wait complete. 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.132+0000 7f48acdf4640 10 monclient: build_initial_monmap 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 10 monclient: monmap: 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:19.134 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 1 librados: starting msgr at 2024-03-25T18:22:19.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 1 librados: starting objecter 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 1 Processor -- start 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 1 -- start start 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 1 librados: setting wanted keys 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 1 librados: calling monclient init 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.133+0000 7f48acdf4640 10 monclient: init 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:19.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient(hunting): picked mon.noname-b con 0x7f48a806e200 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient(hunting): picked mon.noname-a con 0x7f48a8070b50 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient(hunting): picked mon.noname-c con 0x7f48a8152ec0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a807b1c0 con 0x7f48a8070b50 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a807fbe0 con 0x7f48a806e200 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a807fe10 con 0x7f48a8152ec0 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:19.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.134+0000 7f48acdf4640 10 monclient(hunting): authenticate will time out at 7779.397270s 2024-03-25T18:22:19.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.145+0000 7f48a6d9d640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f48a8152ec0 0x7f48a81e4560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:49784/0 (socket says 172.21.15.132:49784) 2024-03-25T18:22:19.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.145+0000 7f48a6d9d640 1 -- 172.21.15.132:0/3443042294 learned_addr learned my addr 172.21.15.132:0/3443042294 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:19.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.145+0000 7f4896ffd640 10 client.?.objecter ms_handle_connect 0x7f48a8152ec0 2024-03-25T18:22:19.147 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.145+0000 7f4896ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:19.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3884013719 0 0) 0x7f48a807fe10 con 0x7f48a8152ec0 2024-03-25T18:22:19.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:19.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:19.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:19.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:22:19.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4888004eb0 con 0x7f48a8152ec0 2024-03-25T18:22:19.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 10 client.?.objecter ms_handle_connect 0x7f48a8070b50 2024-03-25T18:22:19.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.157+0000 7f4896ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:19.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.158+0000 7f4896ffd640 10 client.?.objecter ms_handle_connect 0x7f48a806e200 2024-03-25T18:22:19.159 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.158+0000 7f4896ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:19.175 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.174+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275198454 0 0) 0x7f48a807b1c0 con 0x7f48a8070b50 2024-03-25T18:22:19.175 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.174+0000 7f4896ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:19.175 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.174+0000 7f4896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:19.175 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.174+0000 7f4896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:19.175 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.174+0000 7f4896ffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:22:19.175 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.174+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a807fe10 con 0x7f48a8070b50 2024-03-25T18:22:19.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.181+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183474482 0 0) 0x7f48a807fbe0 con 0x7f48a806e200 2024-03-25T18:22:19.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.181+0000 7f4896ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:19.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.181+0000 7f4896ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:19.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.181+0000 7f4896ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:19.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.181+0000 7f4896ffd640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:22:19.183 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.181+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a807b1c0 con 0x7f48a806e200 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2288931176 0 0) 0x7f4888004eb0 con 0x7f48a8152ec0 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48a807fbe0 con 0x7f48a8152ec0 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f48a0001c90 con 0x7f48a8152ec0 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.185 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:19.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.184+0000 7f4896ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:19.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2334414020 0 0) 0x7f48a807fbe0 con 0x7f48a8152ec0 2024-03-25T18:22:19.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 >> v1:172.21.15.161:6789/0 conn(0x7f48a806e200 legacy=0x7f48a807e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 >> v1:172.21.15.132:6789/0 conn(0x7f48a8070b50 legacy=0x7f48a814ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 1 monclient: found mon.c 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a807da90 con 0x7f48a8152ec0 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f4896ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 10 monclient: _renew_subs 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a814f9a0 con 0x7f48a8152ec0 2024-03-25T18:22:19.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 10 monclient: _renew_subs 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a814f5b0 con 0x7f48a8152ec0 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.206+0000 7f48acdf4640 1 librados: init done 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.207+0000 7f48acdf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.207+0000 7f48acdf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.207+0000 7f48acdf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:19.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.207+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a807ef20 con 0x7f48a8152ec0 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48a0003990 con 0x7f48a8152ec0 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f4896ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f48a0003ce0 con 0x7f48a8152ec0 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f4896ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f4896ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f4896ffd640 10 monclient: dump: 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.466 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.464+0000 7f48a4d99640 4 set_mon_vals no callback set 2024-03-25T18:22:19.467 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.465+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f48a0003f00 con 0x7f48a8152ec0 2024-03-25T18:22:19.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.465+0000 7f48a659c640 10 monclient: get_auth_request con 0x7f488806de20 auth_method 0 2024-03-25T18:22:19.468 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.466+0000 7f4896ffd640 10 client.4613.objecter ms_handle_connect 0x7f488806de20 2024-03-25T18:22:19.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f48a0054ad0 con 0x7f48a8152ec0 2024-03-25T18:22:19.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 10 client.4613.objecter ms_dispatch 0x7f48a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:19.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:19.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:19.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-25T18:22:19.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48a00505e0 con 0x7f48a8152ec0 2024-03-25T18:22:19.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:19.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.470+0000 7f4896ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:19.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.691+0000 7f48acdf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.691+0000 7f48acdf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.691+0000 7f48acdf4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:19.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.691+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48a80707c0 con 0x7f48a8152ec0 2024-03-25T18:22:19.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.741+0000 7f4896ffd640 1 -- 172.21.15.132:0/3443042294 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f48a0052510 con 0x7f48a8152ec0 2024-03-25T18:22:19.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.741+0000 7f4896ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.741+0000 7f4896ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:19.742 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:19.743 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 10 librados: watch_flush enter 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 10 librados: watch_flush exit 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 10 client.4613.objecter successfully canceled tick 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 >> v1:172.21.15.161:6800/411950978 conn(0x7f488806de20 legacy=0x7f48880702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 10 monclient: shutdown 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 >> v1:172.21.15.132:6790/0 conn(0x7f48a8152ec0 legacy=0x7f48a81e4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 shutdown_connections 2024-03-25T18:22:19.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 >> 172.21.15.132:0/3443042294 conn(0x7f48a806b630 msgr2=0x7f48a8071180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:19.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 shutdown_connections 2024-03-25T18:22:19.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.743+0000 7f48acdf4640 1 -- 172.21.15.132:0/3443042294 wait complete. 2024-03-25T18:22:19.746 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:19.744+0000 7f48acdf4640 1 librados: shutdown 2024-03-25T18:22:19.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:26.564 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:26.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.677+0000 7f2471740640 10 monclient: get_monmap_and_config 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.677+0000 7f2471740640 10 monclient: build_initial_monmap 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.677+0000 7f2471740640 10 monclient: monmap: 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:26.678737+0000 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:26.678737+0000 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:26.679 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:26.680 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:26.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.678+0000 7f2471740640 1 Processor -- start 2024-03-25T18:22:26.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.678+0000 7f2471740640 1 -- start start 2024-03-25T18:22:26.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.678+0000 7f2471740640 10 monclient: init 2024-03-25T18:22:26.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:26.681 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient(hunting): picked mon.noname-b con 0x7f246c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient(hunting): picked mon.noname-a con 0x7f246c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient(hunting): picked mon.noname-c con 0x7f246c1430c0 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c142520 con 0x7f246c13f710 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0b28a0 con 0x7f246c1430c0 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0b2a30 con 0x7f246c0a98f0 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:26.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.679+0000 7f2471740640 10 monclient(hunting): authenticate will time out at 7786.942538s 2024-03-25T18:22:26.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.689+0000 7f246b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f246c1430c0 0x7f246c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:51116/0 (socket says 172.21.15.132:51116) 2024-03-25T18:22:26.690 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.689+0000 7f246b7fe640 1 -- 172.21.15.132:0/150770570 learned_addr learned my addr 172.21.15.132:0/150770570 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:26.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2953179169 0 0) 0x7f246c0b28a0 con 0x7f246c1430c0 2024-03-25T18:22:26.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:22:26.704 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454003520 con 0x7f246c1430c0 2024-03-25T18:22:26.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059381522 0 0) 0x7f246c142520 con 0x7f246c13f710 2024-03-25T18:22:26.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:22:26.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.703+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c0b28a0 con 0x7f246c13f710 2024-03-25T18:22:26.710 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.709+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163544574 0 0) 0x7f246c0b2a30 con 0x7f246c0a98f0 2024-03-25T18:22:26.710 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.709+0000 7f2469ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.710 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.709+0000 7f2469ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.710 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.709+0000 7f2469ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.710 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.709+0000 7f2469ffb640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:22:26.710 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.709+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c142520 con 0x7f246c0a98f0 2024-03-25T18:22:26.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167309979 0 0) 0x7f246c0b28a0 con 0x7f246c13f710 2024-03-25T18:22:26.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 >> v1:172.21.15.132:6790/0 conn(0x7f246c0a98f0 legacy=0x7f246c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 >> v1:172.21.15.161:6789/0 conn(0x7f246c1430c0 legacy=0x7f246c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:26.712 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 1 monclient: found mon.noname-a 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c0b44c0 con 0x7f246c13f710 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2460001670 con 0x7f246c13f710 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2469ffb640 10 monclient: dump: 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.713 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2471740640 5 monclient: authenticate success, global_id 4605 2024-03-25T18:22:26.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.711+0000 7f2471740640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24600017d0 con 0x7f246c13f710 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2469ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2469ffb640 1 -- 172.21.15.132:0/150770570 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2460001be0 con 0x7f246c13f710 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2471740640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2471740640 4 set_mon_vals no callback set 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2469ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2469ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.714+0000 7f2469ffb640 10 monclient: dump: 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.716 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:26.717 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:26.717 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:26.717 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:26.717 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:26.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 10 monclient: shutdown 2024-03-25T18:22:26.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:26.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 1 -- 172.21.15.132:0/150770570 >> v1:172.21.15.132:6789/0 conn(0x7f246c13f710 legacy=0x7f246c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 1 -- 172.21.15.132:0/150770570 shutdown_connections 2024-03-25T18:22:26.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 1 -- 172.21.15.132:0/150770570 >> 172.21.15.132:0/150770570 conn(0x7f246c0b7ea0 msgr2=0x7f246c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 1 -- 172.21.15.132:0/150770570 shutdown_connections 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.715+0000 7f2471740640 1 -- 172.21.15.132:0/150770570 wait complete. 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.716+0000 7f2471740640 10 monclient: build_initial_monmap 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.716+0000 7f2471740640 10 monclient: monmap: 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:26.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 1 librados: starting msgr at 2024-03-25T18:22:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 1 librados: starting objecter 2024-03-25T18:22:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 1 Processor -- start 2024-03-25T18:22:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 1 -- start start 2024-03-25T18:22:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 1 librados: setting wanted keys 2024-03-25T18:22:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 1 librados: calling monclient init 2024-03-25T18:22:26.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 10 monclient: init 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.717+0000 7f2471740640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 10 monclient(hunting): picked mon.noname-b con 0x7f246c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 10 monclient(hunting): picked mon.noname-c con 0x7f246c13f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 10 monclient(hunting): picked mon.noname-a con 0x7f246c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c071ce0 con 0x7f246c1430c0 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c06f4d0 con 0x7f246c0a98f0 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c06f660 con 0x7f246c13f710 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:26.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.718+0000 7f2471740640 10 monclient(hunting): authenticate will time out at 7786.981112s 2024-03-25T18:22:26.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.726+0000 7f246affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f246c0a98f0 0x7f246c06edc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:51150/0 (socket says 172.21.15.132:51150) 2024-03-25T18:22:26.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.726+0000 7f246affd640 1 -- 172.21.15.132:0/787640075 learned_addr learned my addr 172.21.15.132:0/787640075 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:26.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.726+0000 7f24537fe640 10 client.?.objecter ms_handle_connect 0x7f246c0a98f0 2024-03-25T18:22:26.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.726+0000 7f24537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44010637 0 0) 0x7f246c06f4d0 con 0x7f246c0a98f0 2024-03-25T18:22:26.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.731 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:22:26.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2458004eb0 con 0x7f246c0a98f0 2024-03-25T18:22:26.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 10 client.?.objecter ms_handle_connect 0x7f246c1430c0 2024-03-25T18:22:26.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.730+0000 7f24537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.739+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256816366 0 0) 0x7f246c071ce0 con 0x7f246c1430c0 2024-03-25T18:22:26.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.739+0000 7f24537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.739+0000 7f24537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.739+0000 7f24537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.739+0000 7f24537fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:22:26.740 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.739+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c06f4d0 con 0x7f246c1430c0 2024-03-25T18:22:26.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.753+0000 7f24537fe640 10 client.?.objecter ms_handle_connect 0x7f246c13f710 2024-03-25T18:22:26.754 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.753+0000 7f24537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.763 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3874558092 0 0) 0x7f2458004eb0 con 0x7f246c0a98f0 2024-03-25T18:22:26.763 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f246c071ce0 con 0x7f246c0a98f0 2024-03-25T18:22:26.763 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f245c001aa0 con 0x7f246c0a98f0 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 10 monclient(hunting): dump: 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:26.764 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:26.765 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.762+0000 7f24537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.765+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1535793824 0 0) 0x7f246c06f4d0 con 0x7f246c1430c0 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.765+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2458004eb0 con 0x7f246c1430c0 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.765+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2464001af0 con 0x7f246c1430c0 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.765+0000 7f24537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.765+0000 7f24537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.765+0000 7f24537fe640 10 monclient(hunting): dump: 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:26.766 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:26.767 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:26.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.767+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169392177 0 0) 0x7f246c06f660 con 0x7f246c13f710 2024-03-25T18:22:26.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.767+0000 7f24537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.768 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.767+0000 7f24537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.767+0000 7f24537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.767+0000 7f24537fe640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:22:26.769 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.767+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c06f4d0 con 0x7f246c13f710 2024-03-25T18:22:26.775 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2118924189 0 0) 0x7f246c071ce0 con 0x7f246c0a98f0 2024-03-25T18:22:26.775 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 >> v1:172.21.15.132:6790/0 conn(0x7f246c13f710 legacy=0x7f246c139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 >> v1:172.21.15.132:6789/0 conn(0x7f246c1430c0 legacy=0x7f246c070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 1 monclient: found mon.b 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c139770 con 0x7f246c0a98f0 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f24537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 10 monclient: _renew_subs 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c139960 con 0x7f246c0a98f0 2024-03-25T18:22:26.776 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 10 client.4627.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 10 monclient: _renew_subs 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f246c06e080 con 0x7f246c0a98f0 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.774+0000 7f2471740640 1 librados: init done 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.775+0000 7f2471740640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.775+0000 7f2471740640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.775+0000 7f2471740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:26.777 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:26.775+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246c071b40 con 0x7f246c0a98f0 2024-03-25T18:22:27.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f245c001d00 con 0x7f246c0a98f0 2024-03-25T18:22:27.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f245c0038c0 con 0x7f246c0a98f0 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24537fe640 10 monclient: dump: 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:27.066 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:27.067 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:27.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.064+0000 7f24697fa640 4 set_mon_vals no callback set 2024-03-25T18:22:27.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.065+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f245c003ae0 con 0x7f246c0a98f0 2024-03-25T18:22:27.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.066+0000 7f246a7fc640 10 monclient: get_auth_request con 0x7f245806df50 auth_method 0 2024-03-25T18:22:27.067 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.066+0000 7f24537fe640 10 client.4627.objecter ms_handle_connect 0x7f245806df50 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.071+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f245c054770 con 0x7f246c0a98f0 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.071+0000 7f24537fe640 10 client.4627.objecter ms_dispatch 0x7f246c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.071+0000 7f24537fe640 3 client.4627.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.071+0000 7f24537fe640 3 client.4627.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.072+0000 7f24537fe640 20 client.4627.objecter dump_active .. 0 homeless 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.072+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f245c050280 con 0x7f246c0a98f0 2024-03-25T18:22:27.073 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.072+0000 7f24537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:27.074 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.072+0000 7f24537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:27.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.316+0000 7f2471740640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:27.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.316+0000 7f2471740640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:27.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.316+0000 7f2471740640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:27.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.316+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f246c0b27a0 con 0x7f246c0a98f0 2024-03-25T18:22:27.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.441+0000 7f24537fe640 1 -- 172.21.15.132:0/787640075 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f245c0521b0 con 0x7f246c0a98f0 2024-03-25T18:22:27.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.441+0000 7f24537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:27.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.441+0000 7f24537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:27.442 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:27.442 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 10 librados: watch_flush enter 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 10 librados: watch_flush exit 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 20 client.4627.objecter shutdown clearing up homeless session... 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 10 client.4627.objecter successfully canceled tick 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 >> v1:172.21.15.161:6800/411950978 conn(0x7f245806df50 legacy=0x7f2458070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 10 monclient: shutdown 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 >> v1:172.21.15.161:6789/0 conn(0x7f246c0a98f0 legacy=0x7f246c06edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 shutdown_connections 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 >> 172.21.15.132:0/787640075 conn(0x7f246c0b7ea0 msgr2=0x7f246c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:27.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 shutdown_connections 2024-03-25T18:22:27.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 -- 172.21.15.132:0/787640075 wait complete. 2024-03-25T18:22:27.446 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:27.443+0000 7f2471740640 1 librados: shutdown 2024-03-25T18:22:27.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:34.265 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:34.379 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.378+0000 7fee672ba640 10 monclient: get_monmap_and_config 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.378+0000 7fee672ba640 10 monclient: build_initial_monmap 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.378+0000 7fee672ba640 10 monclient: monmap: 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:34.379773+0000 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:34.379773+0000 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:34.380 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.379+0000 7fee672ba640 1 Processor -- start 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.379+0000 7fee672ba640 1 -- start start 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.379+0000 7fee672ba640 10 monclient: init 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient(hunting): picked mon.noname-b con 0x7fee600b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient(hunting): picked mon.noname-a con 0x7fee600b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient(hunting): picked mon.noname-c con 0x7fee6014be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:34.382 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600af940 con 0x7fee600b1730 2024-03-25T18:22:34.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600b0a70 con 0x7fee6014be90 2024-03-25T18:22:34.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee600b0d50 con 0x7fee600b4d60 2024-03-25T18:22:34.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:34.383 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.380+0000 7fee672ba640 10 monclient(hunting): authenticate will time out at 7794.642901s 2024-03-25T18:22:34.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.387+0000 7fee65857640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fee6014be90 0x7fee6014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:35628/0 (socket says 172.21.15.132:35628) 2024-03-25T18:22:34.388 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.387+0000 7fee65857640 1 -- 172.21.15.132:0/3661607665 learned_addr learned my addr 172.21.15.132:0/3661607665 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:34.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.396+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075517101 0 0) 0x7fee600af940 con 0x7fee600b1730 2024-03-25T18:22:34.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.396+0000 7fee57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:34.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.396+0000 7fee57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:34.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.396+0000 7fee57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:34.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.396+0000 7fee57fff640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:22:34.398 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.397+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c003520 con 0x7fee600b1730 2024-03-25T18:22:34.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.398+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003976213 0 0) 0x7fee600b0a70 con 0x7fee6014be90 2024-03-25T18:22:34.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.398+0000 7fee57fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:34.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.398+0000 7fee57fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:34.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.398+0000 7fee57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:34.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.398+0000 7fee57fff640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:22:34.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.398+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee600af940 con 0x7fee6014be90 2024-03-25T18:22:34.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3987069442 0 0) 0x7fee4c003520 con 0x7fee600b1730 2024-03-25T18:22:34.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 >> v1:172.21.15.132:6790/0 conn(0x7fee600b4d60 legacy=0x7fee6013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 >> v1:172.21.15.161:6789/0 conn(0x7fee6014be90 legacy=0x7fee6014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 1 monclient: found mon.noname-a 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee600b1030 con 0x7fee600b1730 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee50001690 con 0x7fee600b1730 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee672ba640 5 monclient: authenticate success, global_id 4611 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee672ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee57fff640 10 monclient: dump: 2024-03-25T18:22:34.416 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.413+0000 7fee672ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee500017f0 con 0x7fee600b1730 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee57fff640 1 -- 172.21.15.132:0/3661607665 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee50001c00 con 0x7fee600b1730 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee672ba640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee672ba640 4 set_mon_vals no callback set 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.439 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee57fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.438+0000 7fee57fff640 10 monclient: dump: 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:34.440 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.442 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 10 monclient: shutdown 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 1 -- 172.21.15.132:0/3661607665 >> v1:172.21.15.132:6789/0 conn(0x7fee600b1730 legacy=0x7fee600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 1 -- 172.21.15.132:0/3661607665 shutdown_connections 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 1 -- 172.21.15.132:0/3661607665 >> 172.21.15.132:0/3661607665 conn(0x7fee600617a0 msgr2=0x7fee60061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 1 -- 172.21.15.132:0/3661607665 shutdown_connections 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.439+0000 7fee672ba640 1 -- 172.21.15.132:0/3661607665 wait complete. 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 10 monclient: build_initial_monmap 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 10 monclient: monmap: 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:34.443 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 1 librados: starting msgr at 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 1 librados: starting objecter 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 1 Processor -- start 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 1 -- start start 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 1 librados: setting wanted keys 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 1 librados: calling monclient init 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.440+0000 7fee672ba640 10 monclient: init 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:34.444 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient(hunting): picked mon.noname-a con 0x7fee600b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient(hunting): picked mon.noname-b con 0x7fee600b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient(hunting): picked mon.noname-c con 0x7fee6014be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee60137fa0 con 0x7fee600b1730 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee601361c0 con 0x7fee600b4d60 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee60135ca0 con 0x7fee6014be90 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:34.445 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.441+0000 7fee672ba640 10 monclient(hunting): authenticate will time out at 7794.704268s 2024-03-25T18:22:34.453 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.452+0000 7fee65857640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fee6014be90 0x7fee601e2f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:37700/0 (socket says 172.21.15.132:37700) 2024-03-25T18:22:34.453 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.452+0000 7fee65857640 1 -- 172.21.15.132:0/1385122203 learned_addr learned my addr 172.21.15.132:0/1385122203 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:34.454 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.453+0000 7fee55ffb640 10 client.?.objecter ms_handle_connect 0x7fee6014be90 2024-03-25T18:22:34.454 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.453+0000 7fee55ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:34.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.455+0000 7fee55ffb640 10 client.?.objecter ms_handle_connect 0x7fee600b4d60 2024-03-25T18:22:34.456 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.455+0000 7fee55ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.460+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787322516 0 0) 0x7fee60135ca0 con 0x7fee6014be90 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44004eb0 con 0x7fee6014be90 2024-03-25T18:22:34.462 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 10 client.?.objecter ms_handle_connect 0x7fee600b1730 2024-03-25T18:22:34.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.461+0000 7fee55ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:34.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.467+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843576236 0 0) 0x7fee601361c0 con 0x7fee600b4d60 2024-03-25T18:22:34.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.467+0000 7fee55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:34.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.467+0000 7fee55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:34.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.467+0000 7fee55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:34.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.467+0000 7fee55ffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:22:34.469 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.467+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60135ca0 con 0x7fee600b4d60 2024-03-25T18:22:34.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.470+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790065305 0 0) 0x7fee60137fa0 con 0x7fee600b1730 2024-03-25T18:22:34.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.470+0000 7fee55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:34.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.470+0000 7fee55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:34.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.470+0000 7fee55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:34.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.470+0000 7fee55ffb640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:22:34.471 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.470+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee601361c0 con 0x7fee600b1730 2024-03-25T18:22:34.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 151538327 0 0) 0x7fee44004eb0 con 0x7fee6014be90 2024-03-25T18:22:34.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee60137fa0 con 0x7fee6014be90 2024-03-25T18:22:34.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee58001c10 con 0x7fee6014be90 2024-03-25T18:22:34.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:34.483 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.484 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.482+0000 7fee55ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.495+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3080113401 0 0) 0x7fee60135ca0 con 0x7fee600b4d60 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.495+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee44004eb0 con 0x7fee600b4d60 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.495+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee50001b40 con 0x7fee600b4d60 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.495+0000 7fee55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.495+0000 7fee55ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.495+0000 7fee55ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 836445769 0 0) 0x7fee60137fa0 con 0x7fee6014be90 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 >> v1:172.21.15.161:6789/0 conn(0x7fee600b4d60 legacy=0x7fee601dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 >> v1:172.21.15.132:6789/0 conn(0x7fee600b1730 legacy=0x7fee6012fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 1 monclient: found mon.c 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee60138350 con 0x7fee6014be90 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 5 monclient: authenticate success, global_id 4625 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 10 monclient: _renew_subs 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee60136410 con 0x7fee6014be90 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 10 monclient: _renew_subs 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee60130770 con 0x7fee6014be90 2024-03-25T18:22:34.501 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.498+0000 7fee672ba640 1 librados: init done 2024-03-25T18:22:34.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.499+0000 7fee672ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.499+0000 7fee672ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.499+0000 7fee672ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:34.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.499+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee60135e30 con 0x7fee6014be90 2024-03-25T18:22:34.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee58004b60 con 0x7fee6014be90 2024-03-25T18:22:34.822 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee58004e70 con 0x7fee6014be90 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee55ffb640 10 monclient: dump: 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:34.823 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:34.824 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:34.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.821+0000 7fee577fe640 4 set_mon_vals no callback set 2024-03-25T18:22:34.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.822+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fee5801ae40 con 0x7fee6014be90 2024-03-25T18:22:34.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.822+0000 7fee65056640 10 monclient: get_auth_request con 0x7fee4406df70 auth_method 0 2024-03-25T18:22:34.824 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.823+0000 7fee55ffb640 10 client.4625.objecter ms_handle_connect 0x7fee4406df70 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fee58054a00 con 0x7fee6014be90 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 10 client.4625.objecter ms_dispatch 0x7fee60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee58050340 con 0x7fee6014be90 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:34.828+0000 7fee55ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:35.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.085+0000 7fee672ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:35.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.085+0000 7fee672ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:35.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.085+0000 7fee672ba640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:35.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.085+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee601365a0 con 0x7fee6014be90 2024-03-25T18:22:35.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.112+0000 7fee55ffb640 1 -- 172.21.15.132:0/1385122203 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fee58052440 con 0x7fee6014be90 2024-03-25T18:22:35.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.112+0000 7fee55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:35.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.112+0000 7fee55ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:35.114 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:35.114 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 10 librados: watch_flush enter 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 10 librados: watch_flush exit 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 10 client.4625.objecter successfully canceled tick 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 >> v1:172.21.15.161:6800/411950978 conn(0x7fee4406df70 legacy=0x7fee44070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 10 monclient: shutdown 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:35.117 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 >> v1:172.21.15.132:6790/0 conn(0x7fee6014be90 legacy=0x7fee601e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 shutdown_connections 2024-03-25T18:22:35.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 >> 172.21.15.132:0/1385122203 conn(0x7fee600617a0 msgr2=0x7fee600b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:35.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 shutdown_connections 2024-03-25T18:22:35.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 -- 172.21.15.132:0/1385122203 wait complete. 2024-03-25T18:22:35.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:35.115+0000 7fee672ba640 1 librados: shutdown 2024-03-25T18:22:35.130 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:35.672 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:20.935 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.680 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:20.944 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.718 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:20.982 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.738 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:20.979 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.744 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:20.986 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.747 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:21.011 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.757 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:21.022 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.771 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:21.012 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:35.780 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:21.044 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:41.937 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.047+0000 7f23a98c3640 10 monclient: get_monmap_and_config 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.047+0000 7f23a98c3640 10 monclient: build_initial_monmap 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.048+0000 7f23a98c3640 10 monclient: monmap: 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:42.049074+0000 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:42.049074+0000 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:42.049 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:42.050 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:42.050 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:42.050 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.048+0000 7f23a98c3640 1 Processor -- start 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.048+0000 7f23a98c3640 1 -- start start 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.048+0000 7f23a98c3640 10 monclient: init 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient(hunting): picked mon.noname-c con 0x7f23a40a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient(hunting): picked mon.noname-a con 0x7f23a413f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient(hunting): picked mon.noname-b con 0x7f23a41430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a4142520 con 0x7f23a413f710 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40b28a0 con 0x7f23a40a98f0 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40b2a30 con 0x7f23a41430c0 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:42.051 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.049+0000 7f23a98c3640 10 monclient(hunting): authenticate will time out at 7802.312479s 2024-03-25T18:22:42.057 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.056+0000 7f23a27fc640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f23a413f710 0x7f23a41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:34278/0 (socket says 172.21.15.132:34278) 2024-03-25T18:22:42.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.057+0000 7f23a27fc640 1 -- 172.21.15.132:0/1664781319 learned_addr learned my addr 172.21.15.132:0/1664781319 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.071+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639873231 0 0) 0x7f23a40b2a30 con 0x7f23a41430c0 2024-03-25T18:22:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.071+0000 7f23a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.071+0000 7f23a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.071+0000 7f23a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.071+0000 7f23a1ffb640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:22:42.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.071+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f238c003520 con 0x7f23a41430c0 2024-03-25T18:22:42.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.074+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451675782 0 0) 0x7f23a4142520 con 0x7f23a413f710 2024-03-25T18:22:42.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.074+0000 7f23a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:42.075 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.074+0000 7f23a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:42.076 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.074+0000 7f23a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:42.076 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.074+0000 7f23a1ffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:22:42.076 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.074+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a40b2a30 con 0x7f23a413f710 2024-03-25T18:22:42.081 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.080+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048560942 0 0) 0x7f23a40b28a0 con 0x7f23a40a98f0 2024-03-25T18:22:42.081 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.080+0000 7f23a1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:42.081 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.080+0000 7f23a1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:42.081 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.080+0000 7f23a1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:42.082 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.080+0000 7f23a1ffb640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:22:42.082 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.080+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4142520 con 0x7f23a40a98f0 2024-03-25T18:22:42.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2220285602 0 0) 0x7f23a40b2a30 con 0x7f23a413f710 2024-03-25T18:22:42.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 >> v1:172.21.15.132:6790/0 conn(0x7f23a41430c0 legacy=0x7f23a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 >> v1:172.21.15.161:6789/0 conn(0x7f23a40a98f0 legacy=0x7f23a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.085 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 1 monclient: found mon.noname-a 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40b44c0 con 0x7f23a413f710 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2398001670 con 0x7f23a413f710 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a1ffb640 10 monclient: dump: 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:42.086 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a98c3640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:22:42.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.084+0000 7f23a98c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23980017d0 con 0x7f23a413f710 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a1ffb640 1 -- 172.21.15.132:0/1664781319 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2398001be0 con 0x7f23a413f710 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a98c3640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a98c3640 4 set_mon_vals no callback set 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:42.100 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.099+0000 7f23a1ffb640 10 monclient: dump: 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:42.101 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.102 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 10 monclient: shutdown 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 1 -- 172.21.15.132:0/1664781319 >> v1:172.21.15.132:6789/0 conn(0x7f23a413f710 legacy=0x7f23a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 1 -- 172.21.15.132:0/1664781319 shutdown_connections 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 1 -- 172.21.15.132:0/1664781319 >> 172.21.15.132:0/1664781319 conn(0x7f23a40b7ea0 msgr2=0x7f23a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 1 -- 172.21.15.132:0/1664781319 shutdown_connections 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.100+0000 7f23a98c3640 1 -- 172.21.15.132:0/1664781319 wait complete. 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 10 monclient: build_initial_monmap 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 10 monclient: monmap: 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:42.103 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 1 librados: starting msgr at 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 1 librados: starting objecter 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 1 Processor -- start 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 1 -- start start 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 1 librados: setting wanted keys 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 1 librados: calling monclient init 2024-03-25T18:22:42.104 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.101+0000 7f23a98c3640 10 monclient: init 2024-03-25T18:22:42.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.102+0000 7f23a98c3640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:42.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.102+0000 7f23a98c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:42.105 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.102+0000 7f23a98c3640 10 monclient(hunting): picked mon.noname-c con 0x7f23a40a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.102+0000 7f23a98c3640 10 monclient(hunting): picked mon.noname-b con 0x7f23a413f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.102+0000 7f23a98c3640 10 monclient(hunting): picked mon.noname-a con 0x7f23a41430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.102+0000 7f23a98c3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a4135590 con 0x7f23a41430c0 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.103+0000 7f23a98c3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a41397f0 con 0x7f23a413f710 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.103+0000 7f23a98c3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40ab9e0 con 0x7f23a40a98f0 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.103+0000 7f23a98c3640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:42.106 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.103+0000 7f23a98c3640 10 monclient(hunting): authenticate will time out at 7802.365861s 2024-03-25T18:22:42.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.108+0000 7f23a37fe640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f23a41430c0 0x7f23a4136130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:34322/0 (socket says 172.21.15.132:34322) 2024-03-25T18:22:42.109 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.108+0000 7f23a37fe640 1 -- 172.21.15.132:0/1370159293 learned_addr learned my addr 172.21.15.132:0/1370159293 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:42.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.111+0000 7f238bfff640 10 client.?.objecter ms_handle_connect 0x7f23a41430c0 2024-03-25T18:22:42.113 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.111+0000 7f238bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:42.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.118+0000 7f238bfff640 10 client.?.objecter ms_handle_connect 0x7f23a413f710 2024-03-25T18:22:42.119 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.118+0000 7f238bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:42.124 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.123+0000 7f238bfff640 10 client.?.objecter ms_handle_connect 0x7f23a40a98f0 2024-03-25T18:22:42.125 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.123+0000 7f238bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:42.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.127+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268779078 0 0) 0x7f23a4135590 con 0x7f23a41430c0 2024-03-25T18:22:42.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.127+0000 7f238bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:42.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.127+0000 7f238bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:42.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.127+0000 7f238bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:42.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.127+0000 7f238bfff640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:22:42.128 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.127+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2390004eb0 con 0x7f23a41430c0 2024-03-25T18:22:42.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.130+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554033614 0 0) 0x7f23a40ab9e0 con 0x7f23a40a98f0 2024-03-25T18:22:42.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.130+0000 7f238bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:42.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.130+0000 7f238bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:42.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.130+0000 7f238bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:42.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.130+0000 7f238bfff640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:22:42.132 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.130+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a4135590 con 0x7f23a40a98f0 2024-03-25T18:22:42.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.138+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086440299 0 0) 0x7f23a41397f0 con 0x7f23a413f710 2024-03-25T18:22:42.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.138+0000 7f238bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:42.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.138+0000 7f238bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:42.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.138+0000 7f238bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:42.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.138+0000 7f238bfff640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:22:42.139 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.138+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23a40ab9e0 con 0x7f23a413f710 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4035074461 0 0) 0x7f23a4135590 con 0x7f23a40a98f0 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a41397f0 con 0x7f23a40a98f0 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2394001870 con 0x7f23a40a98f0 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:42.151 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.152 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.150+0000 7f238bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.159+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1592651483 0 0) 0x7f2390004eb0 con 0x7f23a41430c0 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.159+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23a4135590 con 0x7f23a41430c0 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.159+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f239c001c30 con 0x7f23a41430c0 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.159+0000 7f238bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.159+0000 7f238bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.159+0000 7f238bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:42.161 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:42.162 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:42.162 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1616935379 0 0) 0x7f23a41397f0 con 0x7f23a40a98f0 2024-03-25T18:22:42.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 >> v1:172.21.15.161:6789/0 conn(0x7f23a413f710 legacy=0x7f23a4135a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.166 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 >> v1:172.21.15.132:6789/0 conn(0x7f23a41430c0 legacy=0x7f23a4136130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 1 monclient: found mon.c 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a41351f0 con 0x7f23a40a98f0 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f238bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 5 monclient: authenticate success, global_id 4631 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 10 monclient: _renew_subs 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a4137950 con 0x7f23a40a98f0 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:42.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 10 monclient: _renew_subs 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a4138740 con 0x7f23a40a98f0 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.165+0000 7f23a98c3640 1 librados: init done 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.166+0000 7f23a98c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.166+0000 7f23a98c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.166+0000 7f23a98c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:42.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.166+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a40abee0 con 0x7f23a40a98f0 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2394003610 con 0x7f23a40a98f0 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2394001d80 con 0x7f23a40a98f0 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 10 monclient: dump: 2024-03-25T18:22:42.415 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:42.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f23a17fa640 4 set_mon_vals no callback set 2024-03-25T18:22:42.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.414+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f2394003b20 con 0x7f23a40a98f0 2024-03-25T18:22:42.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.415+0000 7f23a27fc640 10 monclient: get_auth_request con 0x7f239006df70 auth_method 0 2024-03-25T18:22:42.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.415+0000 7f238bfff640 10 client.4631.objecter ms_handle_connect 0x7f239006df70 2024-03-25T18:22:42.420 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f2394053590 con 0x7f23a40a98f0 2024-03-25T18:22:42.420 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 10 client.4631.objecter ms_dispatch 0x7f23a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:42.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:42.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:42.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-25T18:22:42.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f239404f0a0 con 0x7f23a40a98f0 2024-03-25T18:22:42.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:42.421 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.419+0000 7f238bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:42.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.655+0000 7f23a98c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.655+0000 7f23a98c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.655+0000 7f23a98c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:42.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.655+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23a414f2d0 con 0x7f23a40a98f0 2024-03-25T18:22:42.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.686+0000 7f238bfff640 1 -- 172.21.15.132:0/1370159293 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f2394050fd0 con 0x7f23a40a98f0 2024-03-25T18:22:42.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.686+0000 7f238bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.686+0000 7f238bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:42.688 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:42.688 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 10 librados: watch_flush enter 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 10 librados: watch_flush exit 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 10 client.4631.objecter successfully canceled tick 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 >> v1:172.21.15.161:6800/411950978 conn(0x7f239006df70 legacy=0x7f2390070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 10 monclient: shutdown 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 >> v1:172.21.15.132:6790/0 conn(0x7f23a40a98f0 legacy=0x7f23a4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 shutdown_connections 2024-03-25T18:22:42.691 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.689+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 >> 172.21.15.132:0/1370159293 conn(0x7f23a40b7ea0 msgr2=0x7f23a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:42.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.690+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 shutdown_connections 2024-03-25T18:22:42.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.690+0000 7f23a98c3640 1 -- 172.21.15.132:0/1370159293 wait complete. 2024-03-25T18:22:42.692 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:42.690+0000 7f23a98c3640 1 librados: shutdown 2024-03-25T18:22:42.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:49.509 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:49.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.624+0000 7f13da7d6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:49.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.624+0000 7f13da7d6640 10 monclient: build_initial_monmap 2024-03-25T18:22:49.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.624+0000 7f13da7d6640 10 monclient: monmap: 2024-03-25T18:22:49.625 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:49.625497+0000 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:49.625497+0000 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:49.626 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.625+0000 7f13da7d6640 1 Processor -- start 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.625+0000 7f13da7d6640 1 -- start start 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.625+0000 7f13da7d6640 10 monclient: init 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f13d40a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f13d413f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f13d41430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:49.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d4142520 con 0x7f13d41430c0 2024-03-25T18:22:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d40b28a0 con 0x7f13d413f710 2024-03-25T18:22:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d40b2a30 con 0x7f13d40a98f0 2024-03-25T18:22:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:49.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.626+0000 7f13da7d6640 10 monclient(hunting): authenticate will time out at 7809.889001s 2024-03-25T18:22:49.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.635+0000 7f13d37fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f13d413f710 0x7f13d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:39954/0 (socket says 172.21.15.132:39954) 2024-03-25T18:22:49.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.635+0000 7f13d37fe640 1 -- 172.21.15.132:0/2337249639 learned_addr learned my addr 172.21.15.132:0/2337249639 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.653+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009181525 0 0) 0x7f13d40b2a30 con 0x7f13d40a98f0 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc003520 con 0x7f13d40a98f0 2024-03-25T18:22:49.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582061277 0 0) 0x7f13d40b28a0 con 0x7f13d413f710 2024-03-25T18:22:49.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:22:49.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.654+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d40b2a30 con 0x7f13d413f710 2024-03-25T18:22:49.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.661+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255365349 0 0) 0x7f13d4142520 con 0x7f13d41430c0 2024-03-25T18:22:49.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.662+0000 7f13d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.662+0000 7f13d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.662+0000 7f13d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.662+0000 7f13d2ffd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:22:49.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.662+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d40b28a0 con 0x7f13d41430c0 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.666+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1406900955 0 0) 0x7f13d40b2a30 con 0x7f13d413f710 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.666+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 >> v1:172.21.15.132:6790/0 conn(0x7f13d40a98f0 legacy=0x7f13d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 >> v1:172.21.15.132:6789/0 conn(0x7f13d41430c0 legacy=0x7f13d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d40b44c0 con 0x7f13d413f710 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:49.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f13c8001670 con 0x7f13d413f710 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13d2ffd640 10 monclient: dump: 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:49.669 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13da7d6640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:22:49.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.667+0000 7f13da7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c80017d0 con 0x7f13d413f710 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13d2ffd640 1 -- 172.21.15.132:0/2337249639 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f13c8001be0 con 0x7f13d413f710 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13d2ffd640 10 monclient: dump: 2024-03-25T18:22:49.682 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13da7d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:49.683 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.681+0000 7f13da7d6640 4 set_mon_vals no callback set 2024-03-25T18:22:49.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 10 monclient: shutdown 2024-03-25T18:22:49.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:49.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 1 -- 172.21.15.132:0/2337249639 >> v1:172.21.15.161:6789/0 conn(0x7f13d413f710 legacy=0x7f13d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 1 -- 172.21.15.132:0/2337249639 shutdown_connections 2024-03-25T18:22:49.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 1 -- 172.21.15.132:0/2337249639 >> 172.21.15.132:0/2337249639 conn(0x7f13d40b7ea0 msgr2=0x7f13d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 1 -- 172.21.15.132:0/2337249639 shutdown_connections 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.682+0000 7f13da7d6640 1 -- 172.21.15.132:0/2337249639 wait complete. 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.683+0000 7f13da7d6640 10 monclient: build_initial_monmap 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.683+0000 7f13da7d6640 10 monclient: monmap: 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.683+0000 7f13da7d6640 1 librados: starting msgr at 2024-03-25T18:22:49.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.683+0000 7f13da7d6640 1 librados: starting objecter 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.684+0000 7f13da7d6640 1 Processor -- start 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.684+0000 7f13da7d6640 1 -- start start 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.684+0000 7f13da7d6640 1 librados: setting wanted keys 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.684+0000 7f13da7d6640 1 librados: calling monclient init 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.684+0000 7f13da7d6640 10 monclient: init 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f13d40a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f13d413f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f13d41430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:49.687 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d413cba0 con 0x7f13d40a98f0 2024-03-25T18:22:49.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d41dc070 con 0x7f13d413f710 2024-03-25T18:22:49.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d413b5f0 con 0x7f13d41430c0 2024-03-25T18:22:49.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:49.688 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.685+0000 7f13da7d6640 10 monclient(hunting): authenticate will time out at 7809.948011s 2024-03-25T18:22:49.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.696+0000 7f13d3fff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f13d40a98f0 0x7f13d4139e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42506/0 (socket says 172.21.15.132:42506) 2024-03-25T18:22:49.697 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.696+0000 7f13d3fff640 1 -- 172.21.15.132:0/4232189653 learned_addr learned my addr 172.21.15.132:0/4232189653 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:49.698 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.697+0000 7f13d0ff9640 10 client.?.objecter ms_handle_connect 0x7f13d40a98f0 2024-03-25T18:22:49.698 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.697+0000 7f13d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.705 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.704+0000 7f13d0ff9640 10 client.?.objecter ms_handle_connect 0x7f13d413f710 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.704+0000 7f13d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.705+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959080823 0 0) 0x7f13d413cba0 con 0x7f13d40a98f0 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.705+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.705+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.705+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.705+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:22:49.706 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.705+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c0004eb0 con 0x7f13d40a98f0 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 10 client.?.objecter ms_handle_connect 0x7f13d41430c0 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162189802 0 0) 0x7f13d41dc070 con 0x7f13d413f710 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:22:49.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.712+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d413cba0 con 0x7f13d413f710 2024-03-25T18:22:49.725 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595922974 0 0) 0x7f13d413cba0 con 0x7f13d413f710 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d41dc070 con 0x7f13d413f710 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f13c8001ae0 con 0x7f13d413f710 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:49.726 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.724+0000 7f13d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.725+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844100078 0 0) 0x7f13d413b5f0 con 0x7f13d41430c0 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.725+0000 7f13d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.725+0000 7f13d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.725+0000 7f13d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.725+0000 7f13d0ff9640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:22:49.727 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.725+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d413cba0 con 0x7f13d41430c0 2024-03-25T18:22:49.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.728+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 244694970 0 0) 0x7f13c0004eb0 con 0x7f13d40a98f0 2024-03-25T18:22:49.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.728+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13d413b5f0 con 0x7f13d40a98f0 2024-03-25T18:22:49.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.728+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f13c4001990 con 0x7f13d40a98f0 2024-03-25T18:22:49.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.728+0000 7f13d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.728+0000 7f13d0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.728+0000 7f13d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:49.730 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3686842209 0 0) 0x7f13d41dc070 con 0x7f13d413f710 2024-03-25T18:22:49.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 >> v1:172.21.15.132:6790/0 conn(0x7f13d41430c0 legacy=0x7f13d41db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 >> v1:172.21.15.132:6789/0 conn(0x7f13d40a98f0 legacy=0x7f13d4139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 1 monclient: found mon.b 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d413b8d0 con 0x7f13d413f710 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 10 monclient: _renew_subs 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d40b7960 con 0x7f13d413f710 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 10 monclient: _renew_subs 2024-03-25T18:22:49.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:49.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d4138e70 con 0x7f13d413f710 2024-03-25T18:22:49.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.742+0000 7f13da7d6640 1 librados: init done 2024-03-25T18:22:49.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.743+0000 7f13da7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.743+0000 7f13da7d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.743+0000 7f13da7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:49.745 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.743+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d414f2d0 con 0x7f13d413f710 2024-03-25T18:22:49.781 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13c8004a90 con 0x7f13d413f710 2024-03-25T18:22:49.781 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:49.781 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f13c8004da0 con 0x7f13d413f710 2024-03-25T18:22:49.781 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d0ff9640 10 monclient: dump: 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:49.782 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.780+0000 7f13d27fc640 4 set_mon_vals no callback set 2024-03-25T18:22:49.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.781+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f13c801ad70 con 0x7f13d413f710 2024-03-25T18:22:49.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.782+0000 7f13d3fff640 10 monclient: get_auth_request con 0x7f13c006dfb0 auth_method 0 2024-03-25T18:22:49.783 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:49.782+0000 7f13d0ff9640 10 client.4645.objecter ms_handle_connect 0x7f13c006dfb0 2024-03-25T18:22:50.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.041+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f13c8054890 con 0x7f13d413f710 2024-03-25T18:22:50.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.041+0000 7f13d0ff9640 10 client.4645.objecter ms_dispatch 0x7f13d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:50.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.041+0000 7f13d0ff9640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:50.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.041+0000 7f13d0ff9640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:50.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.041+0000 7f13d0ff9640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:22:50.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.044+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13c80503a0 con 0x7f13d413f710 2024-03-25T18:22:50.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.044+0000 7f13d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:50.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.044+0000 7f13d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:50.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.289+0000 7f13da7d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:50.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.289+0000 7f13da7d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:50.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.289+0000 7f13da7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:22:50.291 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.289+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13d40627b0 con 0x7f13d413f710 2024-03-25T18:22:50.408 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.406+0000 7f13d0ff9640 1 -- 172.21.15.132:0/4232189653 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f13c80522d0 con 0x7f13d413f710 2024-03-25T18:22:50.408 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.406+0000 7f13d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:50.408 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.407+0000 7f13d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:50.408 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:50.408 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 10 librados: watch_flush enter 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 10 librados: watch_flush exit 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 >> v1:172.21.15.161:6800/411950978 conn(0x7f13c006dfb0 legacy=0x7f13c0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 10 monclient: shutdown 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 >> v1:172.21.15.161:6789/0 conn(0x7f13d413f710 legacy=0x7f13d41db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 shutdown_connections 2024-03-25T18:22:50.411 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 >> 172.21.15.132:0/4232189653 conn(0x7f13d40b7ea0 msgr2=0x7f13d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:50.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 shutdown_connections 2024-03-25T18:22:50.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 -- 172.21.15.132:0/4232189653 wait complete. 2024-03-25T18:22:50.412 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:50.409+0000 7f13da7d6640 1 librados: shutdown 2024-03-25T18:22:50.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:55.675 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:40.938 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.726 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:40.991 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.783 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:09:41.047 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.783 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:41.025 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.787 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:41.028 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.789 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:41.053 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.797 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:41.062 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.821 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:09:41.063 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:55.825 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:09:41.089 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:57.229 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.333+0000 7f93efa4a640 10 monclient: get_monmap_and_config 2024-03-25T18:22:57.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.333+0000 7f93efa4a640 10 monclient: build_initial_monmap 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.333+0000 7f93efa4a640 10 monclient: monmap: 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:22:57.334526+0000 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:22:57.334526+0000 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:22:57.335 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.334+0000 7f93efa4a640 1 Processor -- start 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.334+0000 7f93efa4a640 1 -- start start 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.334+0000 7f93efa4a640 10 monclient: init 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f93e80b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f93e80b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f93e814be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e80af940 con 0x7f93e80b1730 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e80b0a70 con 0x7f93e814be90 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e80b0d50 con 0x7f93e80b4d60 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:57.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.335+0000 7f93efa4a640 10 monclient(hunting): authenticate will time out at 7817.598237s 2024-03-25T18:22:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.345+0000 7f93ecfe5640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f93e80b1730 0x7f93e80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:42598/0 (socket says 172.21.15.132:42598) 2024-03-25T18:22:57.346 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.345+0000 7f93ecfe5640 1 -- 172.21.15.132:0/3784216873 learned_addr learned my addr 172.21.15.132:0/3784216873 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:57.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.353+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291827762 0 0) 0x7f93e80b0a70 con 0x7f93e814be90 2024-03-25T18:22:57.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.353+0000 7f93dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:57.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.353+0000 7f93dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:57.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.353+0000 7f93dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:57.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.353+0000 7f93dffff640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:22:57.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.353+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93d4003520 con 0x7f93e814be90 2024-03-25T18:22:57.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.357+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173730180 0 0) 0x7f93e80af940 con 0x7f93e80b1730 2024-03-25T18:22:57.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.357+0000 7f93dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:57.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.357+0000 7f93dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:57.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.357+0000 7f93dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:57.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.357+0000 7f93dffff640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:22:57.359 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.357+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e80b0a70 con 0x7f93e80b1730 2024-03-25T18:22:57.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.363+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051864237 0 0) 0x7f93e80b0d50 con 0x7f93e80b4d60 2024-03-25T18:22:57.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.363+0000 7f93dffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:57.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.363+0000 7f93dffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:57.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.363+0000 7f93dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:57.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.363+0000 7f93dffff640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:22:57.365 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.363+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e80af940 con 0x7f93e80b4d60 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3628444529 0 0) 0x7f93d4003520 con 0x7f93e814be90 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 >> v1:172.21.15.132:6790/0 conn(0x7f93e80b4d60 legacy=0x7f93e813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 >> v1:172.21.15.132:6789/0 conn(0x7f93e80b1730 legacy=0x7f93e80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 1 monclient: found mon.noname-c 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:22:57.371 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e80b1030 con 0x7f93e814be90 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f93e0001a10 con 0x7f93e814be90 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93efa4a640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93efa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93dffff640 10 monclient: dump: 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.372 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.373 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:57.373 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:57.373 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:57.373 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:57.373 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:57.373 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.369+0000 7f93efa4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:57.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93e0001b70 con 0x7f93e814be90 2024-03-25T18:22:57.399 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93dffff640 1 -- 172.21.15.132:0/3784216873 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f93e0001f80 con 0x7f93e814be90 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93dffff640 10 monclient: dump: 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:57.400 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93efa4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:57.401 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.398+0000 7f93efa4a640 4 set_mon_vals no callback set 2024-03-25T18:22:57.402 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 10 monclient: shutdown 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 1 -- 172.21.15.132:0/3784216873 >> v1:172.21.15.161:6789/0 conn(0x7f93e814be90 legacy=0x7f93e814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 1 -- 172.21.15.132:0/3784216873 shutdown_connections 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 1 -- 172.21.15.132:0/3784216873 >> 172.21.15.132:0/3784216873 conn(0x7f93e80617a0 msgr2=0x7f93e8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 1 -- 172.21.15.132:0/3784216873 shutdown_connections 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.399+0000 7f93efa4a640 1 -- 172.21.15.132:0/3784216873 wait complete. 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.400+0000 7f93efa4a640 10 monclient: build_initial_monmap 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.400+0000 7f93efa4a640 10 monclient: monmap: 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:22:57.403 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.400+0000 7f93efa4a640 1 librados: starting msgr at 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.400+0000 7f93efa4a640 1 librados: starting objecter 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.400+0000 7f93efa4a640 1 Processor -- start 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 1 -- start start 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 1 librados: setting wanted keys 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 1 librados: calling monclient init 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient: init 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient(hunting): picked mon.noname-a con 0x7f93e80b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:22:57.404 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient(hunting): picked mon.noname-b con 0x7f93e80b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient(hunting): picked mon.noname-c con 0x7f93e814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e81d5630 con 0x7f93e80b1730 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e81382a0 con 0x7f93e80b4d60 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93e81d8870 con 0x7f93e814be90 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.401+0000 7f93efa4a640 10 monclient(hunting): authenticate will time out at 7817.664399s 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.404+0000 7f93edfe7640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f93e814be90 0x7f93e813b450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:37250/0 (socket says 172.21.15.132:37250) 2024-03-25T18:22:57.405 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.404+0000 7f93edfe7640 1 -- 172.21.15.132:0/2769843485 learned_addr learned my addr 172.21.15.132:0/2769843485 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:22:57.408 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.407+0000 7f93ddffb640 10 client.?.objecter ms_handle_connect 0x7f93e814be90 2024-03-25T18:22:57.408 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.407+0000 7f93ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:57.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.413+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319941388 0 0) 0x7f93e81d8870 con 0x7f93e814be90 2024-03-25T18:22:57.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.413+0000 7f93ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:57.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.413+0000 7f93ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:57.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.413+0000 7f93ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:57.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.413+0000 7f93ddffb640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:22:57.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.413+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93cc004eb0 con 0x7f93e814be90 2024-03-25T18:22:57.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.416+0000 7f93ddffb640 10 client.?.objecter ms_handle_connect 0x7f93e80b4d60 2024-03-25T18:22:57.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.416+0000 7f93ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:57.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.416+0000 7f93ddffb640 10 client.?.objecter ms_handle_connect 0x7f93e80b1730 2024-03-25T18:22:57.417 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.416+0000 7f93ddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571501887 0 0) 0x7f93cc004eb0 con 0x7f93e814be90 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93e81d8870 con 0x7f93e814be90 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f93e0001f40 con 0x7f93e814be90 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 10 monclient(hunting): dump: 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:57.422 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:57.423 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.421+0000 7f93ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:57.428 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.427+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609608315 0 0) 0x7f93e81382a0 con 0x7f93e80b4d60 2024-03-25T18:22:57.428 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.427+0000 7f93ddffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:57.428 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.427+0000 7f93ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:57.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.427+0000 7f93ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:57.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.427+0000 7f93ddffb640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:22:57.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.427+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93cc004eb0 con 0x7f93e80b4d60 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769880618 0 0) 0x7f93e81d8870 con 0x7f93e814be90 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 >> v1:172.21.15.161:6789/0 conn(0x7f93e80b4d60 legacy=0x7f93e8136c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 >> v1:172.21.15.132:6789/0 conn(0x7f93e80b1730 legacy=0x7f93e81d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 1 monclient: found mon.c 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e81d7c40 con 0x7f93e814be90 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93efa4a640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93efa4a640 10 monclient: _renew_subs 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93efa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.429+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93e8137360 con 0x7f93e814be90 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.430+0000 7f93efa4a640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.430+0000 7f93efa4a640 10 monclient: _renew_subs 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.430+0000 7f93efa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:57.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.430+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93e81367e0 con 0x7f93e814be90 2024-03-25T18:22:57.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.430+0000 7f93efa4a640 1 librados: init done 2024-03-25T18:22:57.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.431+0000 7f93efa4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.431+0000 7f93efa4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.431+0000 7f93efa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:57.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.431+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93e80627b0 con 0x7f93e814be90 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.734+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93e0001510 con 0x7f93e814be90 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.734+0000 7f93ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.734+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f93e0003dc0 con 0x7f93e814be90 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.734+0000 7f93ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.735+0000 7f93ddffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.735+0000 7f93ddffb640 10 monclient: dump: 2024-03-25T18:22:57.736 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:22:57.737 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.734+0000 7f93df7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:57.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.735+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f93e0003fe0 con 0x7f93e814be90 2024-03-25T18:22:57.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.736+0000 7f93ed7e6640 10 monclient: get_auth_request con 0x7f93cc06dd90 auth_method 0 2024-03-25T18:22:57.738 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.736+0000 7f93ddffb640 10 client.4643.objecter ms_handle_connect 0x7f93cc06dd90 2024-03-25T18:22:57.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7f93e0054a90 con 0x7f93e814be90 2024-03-25T18:22:57.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 10 client.4643.objecter ms_dispatch 0x7f93e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:57.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:57.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:57.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T18:22:57.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93e00505a0 con 0x7f93e814be90 2024-03-25T18:22:57.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.742+0000 7f93ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:57.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.998+0000 7f93efa4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.998+0000 7f93efa4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.998+0000 7f93efa4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:22:57.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:57.998+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93e81d7dd0 con 0x7f93e814be90 2024-03-25T18:22:58.035 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.033+0000 7f93ddffb640 1 -- 172.21.15.132:0/2769843485 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7f93e00524d0 con 0x7f93e814be90 2024-03-25T18:22:58.035 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.033+0000 7f93ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:58.035 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.033+0000 7f93ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:58.035 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:22:58.035 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:58.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 10 librados: watch_flush enter 2024-03-25T18:22:58.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 10 librados: watch_flush exit 2024-03-25T18:22:58.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T18:22:58.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 10 client.4643.objecter successfully canceled tick 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 >> v1:172.21.15.161:6800/411950978 conn(0x7f93cc06dd90 legacy=0x7f93cc070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 10 monclient: shutdown 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.036+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 >> v1:172.21.15.132:6790/0 conn(0x7f93e814be90 legacy=0x7f93e813b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.037+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 shutdown_connections 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.037+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 >> 172.21.15.132:0/2769843485 conn(0x7f93e80617a0 msgr2=0x7f93e813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.037+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 shutdown_connections 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.037+0000 7f93efa4a640 1 -- 172.21.15.132:0/2769843485 wait complete. 2024-03-25T18:22:58.039 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:22:58.037+0000 7f93efa4a640 1 librados: shutdown 2024-03-25T18:22:58.050 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:04.857 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:04.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.970+0000 7fe28d67b640 10 monclient: get_monmap_and_config 2024-03-25T18:23:04.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.970+0000 7fe28d67b640 10 monclient: build_initial_monmap 2024-03-25T18:23:04.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.970+0000 7fe28d67b640 10 monclient: monmap: 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:04.971517+0000 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:04.971517+0000 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:04.972 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.971+0000 7fe28d67b640 1 Processor -- start 2024-03-25T18:23:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.971+0000 7fe28d67b640 1 -- start start 2024-03-25T18:23:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.971+0000 7fe28d67b640 10 monclient: init 2024-03-25T18:23:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:04.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2880b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2880b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient(hunting): picked mon.noname-b con 0x7fe28814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2880af940 con 0x7fe2880b1730 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2880b0a70 con 0x7fe2880b4d60 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2880b0d50 con 0x7fe28814be90 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:04.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.972+0000 7fe28d67b640 10 monclient(hunting): authenticate will time out at 7825.235058s 2024-03-25T18:23:04.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.981+0000 7fe286ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe2880b4d60 0x7fe28813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:53260/0 (socket says 172.21.15.132:53260) 2024-03-25T18:23:04.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.981+0000 7fe286ffd640 1 -- 172.21.15.132:0/2757533636 learned_addr learned my addr 172.21.15.132:0/2757533636 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:04.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.990+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984673166 0 0) 0x7fe2880b0d50 con 0x7fe28814be90 2024-03-25T18:23:04.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.990+0000 7fe285ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.990+0000 7fe285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.990+0000 7fe285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.991 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.990+0000 7fe285ffb640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:23:04.992 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:04.990+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270003520 con 0x7fe28814be90 2024-03-25T18:23:05.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.000+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140869331 0 0) 0x7fe2880b0a70 con 0x7fe2880b4d60 2024-03-25T18:23:05.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.000+0000 7fe285ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:05.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.000+0000 7fe285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:05.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.000+0000 7fe285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:05.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.000+0000 7fe285ffb640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:23:05.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.000+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270003840 con 0x7fe2880b4d60 2024-03-25T18:23:05.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.003+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790466557 0 0) 0x7fe2880af940 con 0x7fe2880b1730 2024-03-25T18:23:05.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.003+0000 7fe285ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:05.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.003+0000 7fe285ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:05.004 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.003+0000 7fe285ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:05.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.003+0000 7fe285ffb640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:23:05.005 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.003+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2880b0a70 con 0x7fe2880b1730 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1370051607 0 0) 0x7fe270003520 con 0x7fe28814be90 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 >> v1:172.21.15.161:6789/0 conn(0x7fe2880b4d60 legacy=0x7fe28813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 >> v1:172.21.15.132:6789/0 conn(0x7fe2880b1730 legacy=0x7fe2880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 1 monclient: found mon.noname-b 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:05.007 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2880b1030 con 0x7fe28814be90 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe2800019f0 con 0x7fe28814be90 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe285ffb640 10 monclient: dump: 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.008 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe28d67b640 5 monclient: authenticate success, global_id 4646 2024-03-25T18:23:05.009 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.005+0000 7fe28d67b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:05.017 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe280001b50 con 0x7fe28814be90 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe285ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe285ffb640 1 -- 172.21.15.132:0/2757533636 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe280001f60 con 0x7fe28814be90 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe28d67b640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe28d67b640 4 set_mon_vals no callback set 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe285ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe285ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.016+0000 7fe285ffb640 10 monclient: dump: 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.018 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:05.019 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:05.019 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:05.019 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:05.019 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:05.019 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:05.020 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.017+0000 7fe28d67b640 10 monclient: shutdown 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.017+0000 7fe28d67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.017+0000 7fe28d67b640 1 -- 172.21.15.132:0/2757533636 >> v1:172.21.15.132:6790/0 conn(0x7fe28814be90 legacy=0x7fe28814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.017+0000 7fe28d67b640 1 -- 172.21.15.132:0/2757533636 shutdown_connections 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.017+0000 7fe28d67b640 1 -- 172.21.15.132:0/2757533636 >> 172.21.15.132:0/2757533636 conn(0x7fe2880617a0 msgr2=0x7fe288061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.017+0000 7fe28d67b640 1 -- 172.21.15.132:0/2757533636 shutdown_connections 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.018+0000 7fe28d67b640 1 -- 172.21.15.132:0/2757533636 wait complete. 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.018+0000 7fe28d67b640 10 monclient: build_initial_monmap 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.018+0000 7fe28d67b640 10 monclient: monmap: 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:05.021 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 1 librados: starting msgr at 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 1 librados: starting objecter 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 1 Processor -- start 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 1 -- start start 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 1 librados: setting wanted keys 2024-03-25T18:23:05.022 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 1 librados: calling monclient init 2024-03-25T18:23:05.023 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.019+0000 7fe28d67b640 10 monclient: init 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.020+0000 7fe28d67b640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.020+0000 7fe28d67b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.020+0000 7fe28d67b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2880b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.020+0000 7fe28d67b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2880b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.020+0000 7fe28d67b640 10 monclient(hunting): picked mon.noname-c con 0x7fe28814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.021+0000 7fe28d67b640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe288142060 con 0x7fe2880b4d60 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.021+0000 7fe28d67b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2881409a0 con 0x7fe2880b1730 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.021+0000 7fe28d67b640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe288137cd0 con 0x7fe28814be90 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.021+0000 7fe28d67b640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:05.024 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.021+0000 7fe28d67b640 10 monclient(hunting): authenticate will time out at 7825.283809s 2024-03-25T18:23:05.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.036+0000 7fe2867fc640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe2880b4d60 0x7fe2881340d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54754/0 (socket says 172.21.15.132:54754) 2024-03-25T18:23:05.037 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.036+0000 7fe2867fc640 1 -- 172.21.15.132:0/486631636 learned_addr learned my addr 172.21.15.132:0/486631636 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:05.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.037+0000 7fe26f7fe640 10 client.?.objecter ms_handle_connect 0x7fe2880b4d60 2024-03-25T18:23:05.038 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.037+0000 7fe26f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:05.041 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.040+0000 7fe26f7fe640 10 client.?.objecter ms_handle_connect 0x7fe2880b1730 2024-03-25T18:23:05.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.040+0000 7fe26f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:05.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.041+0000 7fe26f7fe640 10 client.?.objecter ms_handle_connect 0x7fe28814be90 2024-03-25T18:23:05.042 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.041+0000 7fe26f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:05.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.044+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778175312 0 0) 0x7fe288137cd0 con 0x7fe28814be90 2024-03-25T18:23:05.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.044+0000 7fe26f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:05.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.044+0000 7fe26f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:05.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.044+0000 7fe26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:05.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.044+0000 7fe26f7fe640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:23:05.045 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.044+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe274004eb0 con 0x7fe28814be90 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141801191 0 0) 0x7fe274004eb0 con 0x7fe28814be90 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe288137cd0 con 0x7fe28814be90 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe280001e60 con 0x7fe28814be90 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166735818 0 0) 0x7fe2881409a0 con 0x7fe2880b1730 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.056+0000 7fe26f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:05.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.057+0000 7fe26f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:05.060 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.057+0000 7fe26f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:05.060 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.057+0000 7fe26f7fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:23:05.060 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.057+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe274004eb0 con 0x7fe2880b1730 2024-03-25T18:23:05.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369380550 0 0) 0x7fe288137cd0 con 0x7fe28814be90 2024-03-25T18:23:05.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 >> v1:172.21.15.161:6789/0 conn(0x7fe2880b1730 legacy=0x7fe2881375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.064 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 >> v1:172.21.15.132:6789/0 conn(0x7fe2880b4d60 legacy=0x7fe2881340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 1 monclient: found mon.c 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe288140e40 con 0x7fe28814be90 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe26f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 5 monclient: authenticate success, global_id 4649 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 10 monclient: _renew_subs 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2881347e0 con 0x7fe28814be90 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 10 client.4649.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:05.065 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 10 monclient: _renew_subs 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe288134b80 con 0x7fe28814be90 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.063+0000 7fe28d67b640 1 librados: init done 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.064+0000 7fe28d67b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.064+0000 7fe28d67b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.064+0000 7fe28d67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:05.066 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.064+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe288141050 con 0x7fe28814be90 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe280001590 con 0x7fe28814be90 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe26f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe280003dc0 con 0x7fe28814be90 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe26f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe26f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe26f7fe640 10 monclient: dump: 2024-03-25T18:23:05.086 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:05.087 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.085+0000 7fe2857fa640 4 set_mon_vals no callback set 2024-03-25T18:23:05.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.086+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fe280019d90 con 0x7fe28814be90 2024-03-25T18:23:05.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.086+0000 7fe286ffd640 10 monclient: get_auth_request con 0x7fe27406dd10 auth_method 0 2024-03-25T18:23:05.088 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.087+0000 7fe26f7fe640 10 client.4649.objecter ms_handle_connect 0x7fe27406dd10 2024-03-25T18:23:05.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fe280053850 con 0x7fe28814be90 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 10 client.4649.objecter ms_dispatch 0x7fe288000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 3 client.4649.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 3 client.4649.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 20 client.4649.objecter dump_active .. 0 homeless 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe28004f190 con 0x7fe28814be90 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:05.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.314+0000 7fe26f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:05.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.568+0000 7fe28d67b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.568+0000 7fe28d67b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.568+0000 7fe28d67b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:05.570 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.568+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe288141ea0 con 0x7fe28814be90 2024-03-25T18:23:05.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.588+0000 7fe26f7fe640 1 -- 172.21.15.132:0/486631636 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fe280051290 con 0x7fe28814be90 2024-03-25T18:23:05.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.588+0000 7fe26f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.589 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.588+0000 7fe26f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:05.590 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:05.590 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 10 librados: watch_flush enter 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 10 librados: watch_flush exit 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 20 client.4649.objecter shutdown clearing up homeless session... 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 10 client.4649.objecter successfully canceled tick 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 >> v1:172.21.15.161:6800/411950978 conn(0x7fe27406dd10 legacy=0x7fe2740701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 10 monclient: shutdown 2024-03-25T18:23:05.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:05.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 >> v1:172.21.15.132:6790/0 conn(0x7fe28814be90 legacy=0x7fe2881e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 shutdown_connections 2024-03-25T18:23:05.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.590+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 >> 172.21.15.132:0/486631636 conn(0x7fe2880617a0 msgr2=0x7fe2880b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:05.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.591+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 shutdown_connections 2024-03-25T18:23:05.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.591+0000 7fe28d67b640 1 -- 172.21.15.132:0/486631636 wait complete. 2024-03-25T18:23:05.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:05.591+0000 7fe28d67b640 1 librados: shutdown 2024-03-25T18:23:05.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:12.410 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:12.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.525+0000 7fd5670ff640 10 monclient: get_monmap_and_config 2024-03-25T18:23:12.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.525+0000 7fd5670ff640 10 monclient: build_initial_monmap 2024-03-25T18:23:12.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.525+0000 7fd5670ff640 10 monclient: monmap: 2024-03-25T18:23:12.526 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:12.526 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:12.526 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:12.526383+0000 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:12.526383+0000 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:12.527 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:12.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.526+0000 7fd5670ff640 1 Processor -- start 2024-03-25T18:23:12.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.526+0000 7fd5670ff640 1 -- start start 2024-03-25T18:23:12.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.526+0000 7fd5670ff640 10 monclient: init 2024-03-25T18:23:12.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.526+0000 7fd5670ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.526+0000 7fd5670ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.526+0000 7fd5670ff640 10 monclient(hunting): picked mon.noname-c con 0x7fd56006adf0 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 10 monclient(hunting): picked mon.noname-b con 0x7fd5600aee10 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 10 monclient(hunting): picked mon.noname-a con 0x7fd560140860 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600af690 con 0x7fd560140860 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600b0c70 con 0x7fd56006adf0 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600b0f50 con 0x7fd5600aee10 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:12.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.527+0000 7fd5670ff640 10 monclient(hunting): authenticate will time out at 7832.789791s 2024-03-25T18:23:12.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.541+0000 7fd564e9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd56006adf0 0x7fd56013d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:37136/0 (socket says 172.21.15.132:37136) 2024-03-25T18:23:12.543 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.541+0000 7fd564e9b640 1 -- 172.21.15.132:0/3365138290 learned_addr learned my addr 172.21.15.132:0/3365138290 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.561+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432063148 0 0) 0x7fd5600b0c70 con 0x7fd56006adf0 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.561+0000 7fd55f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.561+0000 7fd55f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.561+0000 7fd55f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.561+0000 7fd55f7fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.561+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd550003520 con 0x7fd56006adf0 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.562+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503398935 0 0) 0x7fd5600af690 con 0x7fd560140860 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.562+0000 7fd55f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.562+0000 7fd55f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.562+0000 7fd55f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.562+0000 7fd55f7fe640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:23:12.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.562+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5600b0c70 con 0x7fd560140860 2024-03-25T18:23:12.565 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.564+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658134498 0 0) 0x7fd5600b0f50 con 0x7fd5600aee10 2024-03-25T18:23:12.565 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.564+0000 7fd55f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.566 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.564+0000 7fd55f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.566 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.564+0000 7fd55f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.566 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.564+0000 7fd55f7fe640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:23:12.566 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.564+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5600af690 con 0x7fd5600aee10 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3040871277 0 0) 0x7fd5600af690 con 0x7fd5600aee10 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 >> v1:172.21.15.161:6789/0 conn(0x7fd56006adf0 legacy=0x7fd56013d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 >> v1:172.21.15.132:6789/0 conn(0x7fd560140860 legacy=0x7fd56014b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56013dc40 con 0x7fd5600aee10 2024-03-25T18:23:12.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd54c0015b0 con 0x7fd5600aee10 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd5670ff640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd5670ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd55f7fe640 10 monclient: dump: 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:12.582 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:12.583 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:12.583 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:12.583 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:12.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.579+0000 7fd5670ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:12.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd54c001710 con 0x7fd5600aee10 2024-03-25T18:23:12.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd55f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd55f7fe640 1 -- 172.21.15.132:0/3365138290 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd54c001b20 con 0x7fd5600aee10 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd5670ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd5670ff640 4 set_mon_vals no callback set 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd55f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd55f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.602+0000 7fd55f7fe640 10 monclient: dump: 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:12.604 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:12.605 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:12.605 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 10 monclient: shutdown 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 1 -- 172.21.15.132:0/3365138290 >> v1:172.21.15.132:6790/0 conn(0x7fd5600aee10 legacy=0x7fd560140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 1 -- 172.21.15.132:0/3365138290 shutdown_connections 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 1 -- 172.21.15.132:0/3365138290 >> 172.21.15.132:0/3365138290 conn(0x7fd5600b4a80 msgr2=0x7fd5600b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 1 -- 172.21.15.132:0/3365138290 shutdown_connections 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.603+0000 7fd5670ff640 1 -- 172.21.15.132:0/3365138290 wait complete. 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 10 monclient: build_initial_monmap 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 10 monclient: monmap: 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:12.607 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 1 librados: starting msgr at 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 1 librados: starting objecter 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 1 Processor -- start 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 1 -- start start 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 1 librados: setting wanted keys 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 1 librados: calling monclient init 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.604+0000 7fd5670ff640 10 monclient: init 2024-03-25T18:23:12.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient(hunting): picked mon.noname-a con 0x7fd56006adf0 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient(hunting): picked mon.noname-b con 0x7fd5600aee10 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient(hunting): picked mon.noname-c con 0x7fd56006f450 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600aa7f0 con 0x7fd56006adf0 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600ab8c0 con 0x7fd5600aee10 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5600abba0 con 0x7fd56006f450 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:12.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.605+0000 7fd5670ff640 10 monclient(hunting): authenticate will time out at 7832.868169s 2024-03-25T18:23:12.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.626+0000 7fd56569c640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fd56006f450 0x7fd5601e33e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:41612/0 (socket says 172.21.15.132:41612) 2024-03-25T18:23:12.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.626+0000 7fd56569c640 1 -- 172.21.15.132:0/1148119097 learned_addr learned my addr 172.21.15.132:0/1148119097 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:12.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.631+0000 7fd55d7fa640 10 client.?.objecter ms_handle_connect 0x7fd56006f450 2024-03-25T18:23:12.632 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.631+0000 7fd55d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.633 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.632+0000 7fd55d7fa640 10 client.?.objecter ms_handle_connect 0x7fd56006adf0 2024-03-25T18:23:12.634 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.632+0000 7fd55d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.635+0000 7fd55d7fa640 10 client.?.objecter ms_handle_connect 0x7fd5600aee10 2024-03-25T18:23:12.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.635+0000 7fd55d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.640+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752130376 0 0) 0x7fd5600abba0 con 0x7fd56006f450 2024-03-25T18:23:12.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.640+0000 7fd55d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.640+0000 7fd55d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.640+0000 7fd55d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.640+0000 7fd55d7fa640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:23:12.642 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.640+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544004eb0 con 0x7fd56006f450 2024-03-25T18:23:12.645 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.644+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565736447 0 0) 0x7fd5600aa7f0 con 0x7fd56006adf0 2024-03-25T18:23:12.645 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.644+0000 7fd55d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.646 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.644+0000 7fd55d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.646 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.644+0000 7fd55d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.646 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.644+0000 7fd55d7fa640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:23:12.646 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.644+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5600abba0 con 0x7fd56006adf0 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3000715143 0 0) 0x7fd544004eb0 con 0x7fd56006f450 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5600aa7f0 con 0x7fd56006f450 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd558001c90 con 0x7fd56006f450 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 10 monclient(hunting): dump: 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.650 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:12.651 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.648+0000 7fd55d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:12.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.652+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159124687 0 0) 0x7fd5600ab8c0 con 0x7fd5600aee10 2024-03-25T18:23:12.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.652+0000 7fd55d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.653 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.652+0000 7fd55d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.652+0000 7fd55d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.652+0000 7fd55d7fa640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:23:12.654 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.652+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd544004eb0 con 0x7fd5600aee10 2024-03-25T18:23:12.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2868628976 0 0) 0x7fd5600aa7f0 con 0x7fd56006f450 2024-03-25T18:23:12.655 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 >> v1:172.21.15.161:6789/0 conn(0x7fd5600aee10 legacy=0x7fd56012ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 >> v1:172.21.15.132:6789/0 conn(0x7fd56006adf0 legacy=0x7fd56006ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 1 monclient: found mon.c 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5600aac80 con 0x7fd56006f450 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd55d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 10 monclient: _renew_subs 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5600aa200 con 0x7fd56006f450 2024-03-25T18:23:12.656 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 10 client.4655.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 10 monclient: _renew_subs 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56012b2a0 con 0x7fd56006f450 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.653+0000 7fd5670ff640 1 librados: init done 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.654+0000 7fd5670ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.654+0000 7fd5670ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.654+0000 7fd5670ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:12.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.654+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5600627b0 con 0x7fd56006f450 2024-03-25T18:23:12.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd558003990 con 0x7fd56006f450 2024-03-25T18:23:12.966 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd558003d60 con 0x7fd56006f450 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55d7fa640 10 monclient: dump: 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:12.967 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:12.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.965+0000 7fd55effd640 4 set_mon_vals no callback set 2024-03-25T18:23:12.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.966+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fd558003f80 con 0x7fd56006f450 2024-03-25T18:23:12.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.966+0000 7fd564e9b640 10 monclient: get_auth_request con 0x7fd54406de00 auth_method 0 2024-03-25T18:23:12.968 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.967+0000 7fd55d7fa640 10 client.4655.objecter ms_handle_connect 0x7fd54406de00 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fd558054ad0 con 0x7fd56006f450 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 10 client.4655.objecter ms_dispatch 0x7fd560000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 3 client.4655.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 3 client.4655.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 20 client.4655.objecter dump_active .. 0 homeless 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd5580505e0 con 0x7fd56006f450 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:12.984+0000 7fd55d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:13.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.229+0000 7fd5670ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:13.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.229+0000 7fd5670ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:13.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.229+0000 7fd5670ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:23:13.230 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.229+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5600aff00 con 0x7fd56006f450 2024-03-25T18:23:13.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.282+0000 7fd55d7fa640 1 -- 172.21.15.132:0/1148119097 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fd558052510 con 0x7fd56006f450 2024-03-25T18:23:13.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.282+0000 7fd55d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:13.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.282+0000 7fd55d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:13.284 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:13.284 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:13.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 10 librados: watch_flush enter 2024-03-25T18:23:13.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 10 librados: watch_flush exit 2024-03-25T18:23:13.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 20 client.4655.objecter shutdown clearing up homeless session... 2024-03-25T18:23:13.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 10 client.4655.objecter successfully canceled tick 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 >> v1:172.21.15.161:6800/411950978 conn(0x7fd54406de00 legacy=0x7fd5440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 10 monclient: shutdown 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.284+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 >> v1:172.21.15.132:6790/0 conn(0x7fd56006f450 legacy=0x7fd5601e33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.285+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 shutdown_connections 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.285+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 >> 172.21.15.132:0/1148119097 conn(0x7fd5600b4a80 msgr2=0x7fd560074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.285+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 shutdown_connections 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.285+0000 7fd5670ff640 1 -- 172.21.15.132:0/1148119097 wait complete. 2024-03-25T18:23:13.287 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:13.285+0000 7fd5670ff640 1 librados: shutdown 2024-03-25T18:23:13.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:15.817 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:01.080 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.823 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:01.087 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.854 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:01.118 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.873 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:01.114 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.879 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:01.143 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.880 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:01.121 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.884 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:01.148 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.911 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:01.152 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:15.918 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:01.182 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:20.104 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:20.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.204+0000 7fc97d225640 10 monclient: get_monmap_and_config 2024-03-25T18:23:20.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.204+0000 7fc97d225640 10 monclient: build_initial_monmap 2024-03-25T18:23:20.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.204+0000 7fc97d225640 10 monclient: monmap: 2024-03-25T18:23:20.205 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:20.205 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:20.205465+0000 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:20.205465+0000 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:20.206 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.204+0000 7fc97d225640 1 Processor -- start 2024-03-25T18:23:20.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 1 -- start start 2024-03-25T18:23:20.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient: init 2024-03-25T18:23:20.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:20.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:20.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient(hunting): picked mon.noname-a con 0x7fc9780b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient(hunting): picked mon.noname-b con 0x7fc9780b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient(hunting): picked mon.noname-c con 0x7fc97814be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780af940 con 0x7fc9780b4d60 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780b0a70 con 0x7fc97814be90 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9780b0d50 con 0x7fc9780b1730 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.205+0000 7fc97d225640 10 monclient(hunting): authenticate will time out at 7840.468375s 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.206+0000 7fc97759e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc97814be90 0x7fc97814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41766/0 (socket says 172.21.15.132:41766) 2024-03-25T18:23:20.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.206+0000 7fc97759e640 1 -- 172.21.15.132:0/1496438081 learned_addr learned my addr 172.21.15.132:0/1496438081 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.229+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489220755 0 0) 0x7fc9780af940 con 0x7fc9780b4d60 2024-03-25T18:23:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.229+0000 7fc975d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.229+0000 7fc975d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.229+0000 7fc975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.229+0000 7fc975d9b640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:23:20.231 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.229+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc960003520 con 0x7fc9780b4d60 2024-03-25T18:23:20.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.246+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14447677 0 0) 0x7fc9780b0d50 con 0x7fc9780b1730 2024-03-25T18:23:20.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.246+0000 7fc975d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.246+0000 7fc975d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.246+0000 7fc975d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.246+0000 7fc975d9b640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:23:20.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.246+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9780af940 con 0x7fc9780b1730 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1718903456 0 0) 0x7fc960003520 con 0x7fc9780b4d60 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 >> v1:172.21.15.132:6790/0 conn(0x7fc9780b1730 legacy=0x7fc9780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 >> v1:172.21.15.161:6789/0 conn(0x7fc97814be90 legacy=0x7fc97814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 1 monclient: found mon.noname-a 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9780b1030 con 0x7fc9780b4d60 2024-03-25T18:23:20.256 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: _finish_auth 0 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc968001690 con 0x7fc9780b4d60 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.254+0000 7fc975d9b640 10 monclient: dump: 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:20.257 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.255+0000 7fc97d225640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:23:20.258 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.255+0000 7fc97d225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9680017f0 con 0x7fc9780b4d60 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc975d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc975d9b640 1 -- 172.21.15.132:0/1496438081 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc968001c00 con 0x7fc9780b4d60 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc97d225640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc97d225640 4 set_mon_vals no callback set 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc975d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc975d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc975d9b640 10 monclient: dump: 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:20.279 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:20.280 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc97d225640 10 monclient: shutdown 2024-03-25T18:23:20.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc97d225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:20.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.278+0000 7fc97d225640 1 -- 172.21.15.132:0/1496438081 >> v1:172.21.15.132:6789/0 conn(0x7fc9780b4d60 legacy=0x7fc97813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.279+0000 7fc97d225640 1 -- 172.21.15.132:0/1496438081 shutdown_connections 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.279+0000 7fc97d225640 1 -- 172.21.15.132:0/1496438081 >> 172.21.15.132:0/1496438081 conn(0x7fc9780617a0 msgr2=0x7fc978061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.279+0000 7fc97d225640 1 -- 172.21.15.132:0/1496438081 shutdown_connections 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.279+0000 7fc97d225640 1 -- 172.21.15.132:0/1496438081 wait complete. 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.280+0000 7fc97d225640 10 monclient: build_initial_monmap 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.280+0000 7fc97d225640 10 monclient: monmap: 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:20.282 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:20.283 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.280+0000 7fc97d225640 1 librados: starting msgr at 2024-03-25T18:23:20.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.280+0000 7fc97d225640 1 librados: starting objecter 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.280+0000 7fc97d225640 1 Processor -- start 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 1 -- start start 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 1 librados: setting wanted keys 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 1 librados: calling monclient init 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient: init 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient(hunting): picked mon.noname-c con 0x7fc9780b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient(hunting): picked mon.noname-a con 0x7fc9780b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient(hunting): picked mon.noname-b con 0x7fc97814be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc978142030 con 0x7fc9780b4d60 2024-03-25T18:23:20.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc97813bdf0 con 0x7fc97814be90 2024-03-25T18:23:20.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9781424c0 con 0x7fc9780b1730 2024-03-25T18:23:20.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:20.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.281+0000 7fc97d225640 10 monclient(hunting): authenticate will time out at 7840.544423s 2024-03-25T18:23:20.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.293+0000 7fc97759e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc97814be90 0x7fc978142cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41806/0 (socket says 172.21.15.132:41806) 2024-03-25T18:23:20.294 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.293+0000 7fc97759e640 1 -- 172.21.15.132:0/3828682043 learned_addr learned my addr 172.21.15.132:0/3828682043 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:20.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.296+0000 7fc95f7fe640 10 client.?.objecter ms_handle_connect 0x7fc97814be90 2024-03-25T18:23:20.298 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.296+0000 7fc95f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:20.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.299+0000 7fc95f7fe640 10 client.?.objecter ms_handle_connect 0x7fc9780b1730 2024-03-25T18:23:20.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.299+0000 7fc95f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:20.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.300+0000 7fc95f7fe640 10 client.?.objecter ms_handle_connect 0x7fc9780b4d60 2024-03-25T18:23:20.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.300+0000 7fc95f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:20.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.314+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597810716 0 0) 0x7fc97813bdf0 con 0x7fc97814be90 2024-03-25T18:23:20.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.314+0000 7fc95f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.314+0000 7fc95f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.314+0000 7fc95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.314+0000 7fc95f7fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:23:20.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.314+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc964004eb0 con 0x7fc97814be90 2024-03-25T18:23:20.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.316+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052406927 0 0) 0x7fc978142030 con 0x7fc9780b4d60 2024-03-25T18:23:20.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.316+0000 7fc95f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.316+0000 7fc95f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.317 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.316+0000 7fc95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.316+0000 7fc95f7fe640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:23:20.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.316+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc97813bdf0 con 0x7fc9780b4d60 2024-03-25T18:23:20.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.324+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103442357 0 0) 0x7fc9781424c0 con 0x7fc9780b1730 2024-03-25T18:23:20.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.324+0000 7fc95f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:20.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.324+0000 7fc95f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:20.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.324+0000 7fc95f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:20.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.324+0000 7fc95f7fe640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:23:20.326 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.324+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc978142030 con 0x7fc9780b1730 2024-03-25T18:23:20.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3900318568 0 0) 0x7fc97813bdf0 con 0x7fc9780b4d60 2024-03-25T18:23:20.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9781424c0 con 0x7fc9780b4d60 2024-03-25T18:23:20.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc96c0019f0 con 0x7fc9780b4d60 2024-03-25T18:23:20.334 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.333+0000 7fc95f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.335+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3108548079 0 0) 0x7fc964004eb0 con 0x7fc97814be90 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.335+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc97813bdf0 con 0x7fc97814be90 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.335+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc970001b10 con 0x7fc97814be90 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.335+0000 7fc95f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.335+0000 7fc95f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.335+0000 7fc95f7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:20.336 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:20.337 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.353 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.352+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 157283926 0 0) 0x7fc978142030 con 0x7fc9780b1730 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.352+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc964004eb0 con 0x7fc9780b1730 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.352+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc968001b00 con 0x7fc9780b1730 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.352+0000 7fc95f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.352+0000 7fc95f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.352+0000 7fc95f7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:20.354 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:20.355 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:20.355 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4028295426 0 0) 0x7fc9781424c0 con 0x7fc9780b4d60 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 >> v1:172.21.15.132:6790/0 conn(0x7fc9780b1730 legacy=0x7fc978140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 >> v1:172.21.15.161:6789/0 conn(0x7fc97814be90 legacy=0x7fc978142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 1 monclient: found mon.a 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:20.356 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978142650 con 0x7fc9780b4d60 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc95f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 5 monclient: authenticate success, global_id 4647 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 10 monclient: _renew_subs 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9781db820 con 0x7fc9780b4d60 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 10 client.4647.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 10 monclient: _renew_subs 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9781435a0 con 0x7fc9780b4d60 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.353+0000 7fc97d225640 1 librados: init done 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.354+0000 7fc97d225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.354+0000 7fc97d225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.354+0000 7fc97d225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.354+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9781daf60 con 0x7fc9780b4d60 2024-03-25T18:23:20.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc96c003610 con 0x7fc9780b4d60 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc95f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fc96c003920 con 0x7fc9780b4d60 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc95f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc95f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc95f7fe640 10 monclient: dump: 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:20.658 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:20.659 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:20.659 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:20.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.656+0000 7fc97559a640 4 set_mon_vals no callback set 2024-03-25T18:23:20.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.657+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fc96c0198f0 con 0x7fc9780b4d60 2024-03-25T18:23:20.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.658+0000 7fc976d9d640 10 monclient: get_auth_request con 0x7fc96406e0d0 auth_method 0 2024-03-25T18:23:20.659 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.658+0000 7fc95f7fe640 10 client.4647.objecter ms_handle_connect 0x7fc96406e0d0 2024-03-25T18:23:20.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fc96c0535e0 con 0x7fc9780b4d60 2024-03-25T18:23:20.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 10 client.4647.objecter ms_dispatch 0x7fc978000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:20.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 3 client.4647.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:20.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 3 client.4647.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:20.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 20 client.4647.objecter dump_active .. 0 homeless 2024-03-25T18:23:20.667 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc96c04ef20 con 0x7fc9780b4d60 2024-03-25T18:23:20.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.666+0000 7fc95f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:20.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.915+0000 7fc97d225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.915+0000 7fc97d225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.915+0000 7fc97d225640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:20.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.915+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9781401f0 con 0x7fc9780b4d60 2024-03-25T18:23:20.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.947+0000 7fc95f7fe640 1 -- 172.21.15.132:0/3828682043 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fc96c051020 con 0x7fc9780b4d60 2024-03-25T18:23:20.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.947+0000 7fc95f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.947+0000 7fc95f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:20.949 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:20.949 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:20.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.949+0000 7fc97d225640 10 librados: watch_flush enter 2024-03-25T18:23:20.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.949+0000 7fc97d225640 10 librados: watch_flush exit 2024-03-25T18:23:20.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.949+0000 7fc97d225640 20 client.4647.objecter shutdown clearing up homeless session... 2024-03-25T18:23:20.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.949+0000 7fc97d225640 10 client.4647.objecter successfully canceled tick 2024-03-25T18:23:20.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 >> v1:172.21.15.161:6800/411950978 conn(0x7fc96406e0d0 legacy=0x7fc964070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.951 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 10 monclient: shutdown 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 >> v1:172.21.15.132:6789/0 conn(0x7fc9780b4d60 legacy=0x7fc9781db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 shutdown_connections 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 >> 172.21.15.132:0/3828682043 conn(0x7fc9780617a0 msgr2=0x7fc97813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 shutdown_connections 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 -- 172.21.15.132:0/3828682043 wait complete. 2024-03-25T18:23:20.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:20.950+0000 7fc97d225640 1 librados: shutdown 2024-03-25T18:23:20.963 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:27.770 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.884+0000 7fd7c27a3640 10 monclient: get_monmap_and_config 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.884+0000 7fd7c27a3640 10 monclient: build_initial_monmap 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.884+0000 7fd7c27a3640 10 monclient: monmap: 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:27.885191+0000 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:27.885191+0000 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:27.885 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:27.886 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:27.886 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.885+0000 7fd7c27a3640 1 Processor -- start 2024-03-25T18:23:27.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.885+0000 7fd7c27a3640 1 -- start start 2024-03-25T18:23:27.886 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.885+0000 7fd7c27a3640 10 monclient: init 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient(hunting): picked mon.noname-c con 0x7fd7bc0b4f10 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient(hunting): picked mon.noname-b con 0x7fd7bc0b44d0 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient(hunting): picked mon.noname-a con 0x7fd7bc13f740 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0b0140 con 0x7fd7bc13f740 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0af3a0 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc0af530 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:27.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.886+0000 7fd7c27a3640 10 monclient(hunting): authenticate will time out at 7848.149051s 2024-03-25T18:23:27.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.893+0000 7fd7bbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd7bc0b4f10 0x7fd7bc0ae910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:58010/0 (socket says 172.21.15.132:58010) 2024-03-25T18:23:27.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.893+0000 7fd7bbfff640 1 -- 172.21.15.132:0/2557283145 learned_addr learned my addr 172.21.15.132:0/2557283145 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:27.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.902+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643263884 0 0) 0x7fd7bc0af3a0 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.902+0000 7fd7baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.902+0000 7fd7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.902+0000 7fd7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.902+0000 7fd7baffd640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:23:27.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.902+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a4003520 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.904+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477068082 0 0) 0x7fd7bc0af530 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.904+0000 7fd7baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.904+0000 7fd7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.905 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.904+0000 7fd7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.904+0000 7fd7baffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:23:27.906 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.904+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7bc0af3a0 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.906+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660915463 0 0) 0x7fd7bc0b0140 con 0x7fd7bc13f740 2024-03-25T18:23:27.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.906+0000 7fd7baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.906+0000 7fd7baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.906+0000 7fd7baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.906+0000 7fd7baffd640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:23:27.907 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.906+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7bc0af530 con 0x7fd7bc13f740 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2409857492 0 0) 0x7fd7a4003520 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 >> v1:172.21.15.132:6790/0 conn(0x7fd7bc0b44d0 legacy=0x7fd7bc0b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 >> v1:172.21.15.132:6789/0 conn(0x7fd7bc13f740 legacy=0x7fd7bc13fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 1 monclient: found mon.noname-c 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0afcd0 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:27.913 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd7ac001690 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7c27a3640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7baffd640 10 monclient: dump: 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.911+0000 7fd7c27a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7ac0018b0 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7baffd640 1 -- 172.21.15.132:0/2557283145 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd7ac001c40 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7c27a3640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7c27a3640 4 set_mon_vals no callback set 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7baffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.923+0000 7fd7baffd640 10 monclient: dump: 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:27.925 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 10 monclient: shutdown 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 1 -- 172.21.15.132:0/2557283145 >> v1:172.21.15.161:6789/0 conn(0x7fd7bc0b4f10 legacy=0x7fd7bc0ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 1 -- 172.21.15.132:0/2557283145 shutdown_connections 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 1 -- 172.21.15.132:0/2557283145 >> 172.21.15.132:0/2557283145 conn(0x7fd7bc0617a0 msgr2=0x7fd7bc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 1 -- 172.21.15.132:0/2557283145 shutdown_connections 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.924+0000 7fd7c27a3640 1 -- 172.21.15.132:0/2557283145 wait complete. 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 10 monclient: build_initial_monmap 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 10 monclient: monmap: 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:27.927 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 1 librados: starting msgr at 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 1 librados: starting objecter 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 1 Processor -- start 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 1 -- start start 2024-03-25T18:23:27.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 1 librados: setting wanted keys 2024-03-25T18:23:27.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 1 librados: calling monclient init 2024-03-25T18:23:27.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.925+0000 7fd7c27a3640 10 monclient: init 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.926+0000 7fd7c27a3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.926+0000 7fd7c27a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 10 monclient(hunting): picked mon.noname-a con 0x7fd7bc0b44d0 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 10 monclient(hunting): picked mon.noname-c con 0x7fd7bc0b4f10 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 10 monclient(hunting): picked mon.noname-b con 0x7fd7bc13f740 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc138100 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc1dac80 con 0x7fd7bc13f740 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7bc136820 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:27.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.927+0000 7fd7c27a3640 10 monclient(hunting): authenticate will time out at 7848.190453s 2024-03-25T18:23:27.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.936+0000 7fd7bbfff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fd7bc0b44d0 0x7fd7bc136110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:50888/0 (socket says 172.21.15.132:50888) 2024-03-25T18:23:27.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.936+0000 7fd7bbfff640 1 -- 172.21.15.132:0/3124238477 learned_addr learned my addr 172.21.15.132:0/3124238477 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:27.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.937+0000 7fd7b8ff9640 10 client.?.objecter ms_handle_connect 0x7fd7bc0b44d0 2024-03-25T18:23:27.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.937+0000 7fd7b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.939+0000 7fd7b8ff9640 10 client.?.objecter ms_handle_connect 0x7fd7bc13f740 2024-03-25T18:23:27.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.939+0000 7fd7b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.952+0000 7fd7b8ff9640 10 client.?.objecter ms_handle_connect 0x7fd7bc0b4f10 2024-03-25T18:23:27.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.952+0000 7fd7b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170036482 0 0) 0x7fd7bc1dac80 con 0x7fd7bc13f740 2024-03-25T18:23:27.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a8004eb0 con 0x7fd7bc13f740 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66024603 0 0) 0x7fd7bc138100 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:23:27.955 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.953+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7bc1dac80 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.961+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248296079 0 0) 0x7fd7bc136820 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.961+0000 7fd7b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.961+0000 7fd7b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.961+0000 7fd7b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.961+0000 7fd7b8ff9640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:23:27.962 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.961+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7bc138100 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.975 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 788507115 0 0) 0x7fd7bc1dac80 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7bc136820 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd7ac0015d0 con 0x7fd7bc0b44d0 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.974+0000 7fd7b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.979+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1970885152 0 0) 0x7fd7bc138100 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.979+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7bc1dac80 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.979+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd7b00018f0 con 0x7fd7bc0b4f10 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.979+0000 7fd7b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.979+0000 7fd7b8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.979+0000 7fd7b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.981 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:27.982 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:27.982 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:27.982 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:27.982 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.980+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2931016628 0 0) 0x7fd7a8004eb0 con 0x7fd7bc13f740 2024-03-25T18:23:27.982 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.980+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7bc138100 con 0x7fd7bc13f740 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.980+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd7b4001bd0 con 0x7fd7bc13f740 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.980+0000 7fd7b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.980+0000 7fd7b8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.980+0000 7fd7b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:27.983 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:27.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.992+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2293252043 0 0) 0x7fd7bc138100 con 0x7fd7bc13f740 2024-03-25T18:23:27.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 >> v1:172.21.15.132:6790/0 conn(0x7fd7bc0b4f10 legacy=0x7fd7bc1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 >> v1:172.21.15.132:6789/0 conn(0x7fd7bc0b44d0 legacy=0x7fd7bc136110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 1 monclient: found mon.b 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc1d41a0 con 0x7fd7bc13f740 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 10 monclient: _renew_subs 2024-03-25T18:23:27.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7bc1db8b0 con 0x7fd7bc13f740 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 10 monclient: _renew_subs 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7bc137230 con 0x7fd7bc13f740 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.993+0000 7fd7c27a3640 1 librados: init done 2024-03-25T18:23:27.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.994+0000 7fd7c27a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.994+0000 7fd7c27a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.994+0000 7fd7c27a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:27.997 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:27.994+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7bc1dae10 con 0x7fd7bc13f740 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7b4002060 con 0x7fd7bc13f740 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fd7b4003bb0 con 0x7fd7bc13f740 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7b8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7b8ff9640 10 monclient: dump: 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:28.273 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:28.274 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:28.274 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:28.274 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:28.274 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:28.274 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:28.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.271+0000 7fd7ba7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:28.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.272+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fd7b4019b80 con 0x7fd7bc13f740 2024-03-25T18:23:28.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.272+0000 7fd7bbfff640 10 monclient: get_auth_request con 0x7fd7a806e110 auth_method 0 2024-03-25T18:23:28.275 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.273+0000 7fd7b8ff9640 10 client.4675.objecter ms_handle_connect 0x7fd7a806e110 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.276+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fd7b40538f0 con 0x7fd7bc13f740 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.276+0000 7fd7b8ff9640 10 client.4675.objecter ms_dispatch 0x7fd7bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.276+0000 7fd7b8ff9640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.276+0000 7fd7b8ff9640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.276+0000 7fd7b8ff9640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.277+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7b404f230 con 0x7fd7bc13f740 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.277+0000 7fd7b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:28.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.277+0000 7fd7b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:28.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.501+0000 7fd7c27a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:28.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.501+0000 7fd7c27a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:28.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.501+0000 7fd7c27a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:28.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.501+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7bc1d4330 con 0x7fd7bc13f740 2024-03-25T18:23:28.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.536+0000 7fd7b8ff9640 1 -- 172.21.15.132:0/3124238477 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1831712706) 0x7fd7b4051330 con 0x7fd7bc13f740 2024-03-25T18:23:28.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.537+0000 7fd7b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:28.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.537+0000 7fd7b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:28.538 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:28.538 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":10,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:13:37.303205+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:13:37.303205+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 10 librados: watch_flush enter 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 10 librados: watch_flush exit 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 >> v1:172.21.15.161:6800/411950978 conn(0x7fd7a806e110 legacy=0x7fd7a80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 10 monclient: shutdown 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 >> v1:172.21.15.161:6789/0 conn(0x7fd7bc13f740 legacy=0x7fd7bc1d4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 shutdown_connections 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 >> 172.21.15.132:0/3124238477 conn(0x7fd7bc0617a0 msgr2=0x7fd7bc13e710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:28.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 shutdown_connections 2024-03-25T18:23:28.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 -- 172.21.15.132:0/3124238477 wait complete. 2024-03-25T18:23:28.542 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:28.539+0000 7fd7c27a3640 1 librados: shutdown 2024-03-25T18:23:28.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:35.358 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:35.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.476+0000 7fea43ad6640 10 monclient: get_monmap_and_config 2024-03-25T18:23:35.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.476+0000 7fea43ad6640 10 monclient: build_initial_monmap 2024-03-25T18:23:35.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.477+0000 7fea43ad6640 10 monclient: monmap: 2024-03-25T18:23:35.478 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:35.478079+0000 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:35.478079+0000 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.479 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.477+0000 7fea43ad6640 1 Processor -- start 2024-03-25T18:23:35.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.477+0000 7fea43ad6640 1 -- start start 2024-03-25T18:23:35.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.477+0000 7fea43ad6640 10 monclient: init 2024-03-25T18:23:35.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:35.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:35.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient(hunting): picked mon.noname-a con 0x7fea3c0a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:35.480 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient(hunting): picked mon.noname-c con 0x7fea3c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:35.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient(hunting): picked mon.noname-b con 0x7fea3c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:35.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c142520 con 0x7fea3c0a98f0 2024-03-25T18:23:35.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c0b28a0 con 0x7fea3c13f710 2024-03-25T18:23:35.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c0b2a30 con 0x7fea3c1430c0 2024-03-25T18:23:35.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:35.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.478+0000 7fea43ad6640 10 monclient(hunting): authenticate will time out at 7855.741412s 2024-03-25T18:23:35.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.487+0000 7fea41872640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fea3c0a98f0 0x7fea3c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:57518/0 (socket says 172.21.15.132:57518) 2024-03-25T18:23:35.488 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.487+0000 7fea41872640 1 -- 172.21.15.132:0/2781241218 learned_addr learned my addr 172.21.15.132:0/2781241218 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:35.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.505+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319207691 0 0) 0x7fea3c0b2a30 con 0x7fea3c1430c0 2024-03-25T18:23:35.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.505+0000 7fea40870640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.505+0000 7fea40870640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.505+0000 7fea40870640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.505+0000 7fea40870640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:23:35.506 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.505+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea28003520 con 0x7fea3c1430c0 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.508+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021216267 0 0) 0x7fea3c0b28a0 con 0x7fea3c13f710 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.508+0000 7fea40870640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.508+0000 7fea40870640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.508+0000 7fea40870640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.508+0000 7fea40870640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.508+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c0b2a30 con 0x7fea3c13f710 2024-03-25T18:23:35.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.509+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199466128 0 0) 0x7fea3c142520 con 0x7fea3c0a98f0 2024-03-25T18:23:35.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.509+0000 7fea40870640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.509+0000 7fea40870640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.509+0000 7fea40870640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.509+0000 7fea40870640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:23:35.510 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.509+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c0b28a0 con 0x7fea3c0a98f0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3489703232 0 0) 0x7fea3c0b28a0 con 0x7fea3c0a98f0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 >> v1:172.21.15.132:6790/0 conn(0x7fea3c1430c0 legacy=0x7fea3c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 >> v1:172.21.15.161:6789/0 conn(0x7fea3c13f710 legacy=0x7fea3c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: _finish_hunting 0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 1 monclient: found mon.noname-a 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0b44c0 con 0x7fea3c0a98f0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: _finish_auth 0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: _check_auth_tickets 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fea380015b0 con 0x7fea3c0a98f0 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea40870640 10 monclient: dump: 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea43ad6640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:23:35.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.518+0000 7fea43ad6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:35.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea38001710 con 0x7fea3c0a98f0 2024-03-25T18:23:35.535 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea40870640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea40870640 1 -- 172.21.15.132:0/2781241218 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fea38001b60 con 0x7fea3c0a98f0 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea43ad6640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea43ad6640 4 set_mon_vals no callback set 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea40870640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea40870640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.534+0000 7fea40870640 10 monclient: dump: 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:35.536 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:35.537 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 10 monclient: shutdown 2024-03-25T18:23:35.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:35.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 1 -- 172.21.15.132:0/2781241218 >> v1:172.21.15.132:6789/0 conn(0x7fea3c0a98f0 legacy=0x7fea3c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 1 -- 172.21.15.132:0/2781241218 shutdown_connections 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 1 -- 172.21.15.132:0/2781241218 >> 172.21.15.132:0/2781241218 conn(0x7fea3c0b7ea0 msgr2=0x7fea3c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 1 -- 172.21.15.132:0/2781241218 shutdown_connections 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.535+0000 7fea43ad6640 1 -- 172.21.15.132:0/2781241218 wait complete. 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.536+0000 7fea43ad6640 10 monclient: build_initial_monmap 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.536+0000 7fea43ad6640 10 monclient: monmap: 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:35.539 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.536+0000 7fea43ad6640 1 librados: starting msgr at 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.536+0000 7fea43ad6640 1 librados: starting objecter 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.536+0000 7fea43ad6640 1 Processor -- start 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 1 -- start start 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 1 librados: setting wanted keys 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 1 librados: calling monclient init 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient: init 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient(hunting): picked mon.noname-b con 0x7fea3c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient(hunting): picked mon.noname-a con 0x7fea3c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient(hunting): picked mon.noname-c con 0x7fea3c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c136230 con 0x7fea3c13f710 2024-03-25T18:23:35.540 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c136eb0 con 0x7fea3c0a98f0 2024-03-25T18:23:35.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c137190 con 0x7fea3c1430c0 2024-03-25T18:23:35.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:35.541 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.537+0000 7fea43ad6640 10 monclient(hunting): authenticate will time out at 7855.800381s 2024-03-25T18:23:35.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.546+0000 7fea42073640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7fea3c1430c0 0x7fea3c1db4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:56520/0 (socket says 172.21.15.132:56520) 2024-03-25T18:23:35.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.546+0000 7fea42073640 1 -- 172.21.15.132:0/2925260738 learned_addr learned my addr 172.21.15.132:0/2925260738 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:35.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.547+0000 7fea327fc640 10 client.?.objecter ms_handle_connect 0x7fea3c1430c0 2024-03-25T18:23:35.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.547+0000 7fea327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:35.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.550+0000 7fea327fc640 10 client.?.objecter ms_handle_connect 0x7fea3c0a98f0 2024-03-25T18:23:35.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.550+0000 7fea327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:35.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.551+0000 7fea327fc640 10 client.?.objecter ms_handle_connect 0x7fea3c13f710 2024-03-25T18:23:35.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.551+0000 7fea327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:35.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.562+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737478993 0 0) 0x7fea3c136230 con 0x7fea3c13f710 2024-03-25T18:23:35.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.562+0000 7fea327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.562+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.562+0000 7fea327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.562+0000 7fea327fc640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:23:35.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.562+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea20004eb0 con 0x7fea3c13f710 2024-03-25T18:23:35.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.571+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505422068 0 0) 0x7fea3c136eb0 con 0x7fea3c0a98f0 2024-03-25T18:23:35.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.571+0000 7fea327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.571+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.571+0000 7fea327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.571+0000 7fea327fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:23:35.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.571+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c136230 con 0x7fea3c0a98f0 2024-03-25T18:23:35.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.574+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134845251 0 0) 0x7fea3c137190 con 0x7fea3c1430c0 2024-03-25T18:23:35.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.574+0000 7fea327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:35.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.574+0000 7fea327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:35.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.574+0000 7fea327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:35.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.574+0000 7fea327fc640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:23:35.575 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.574+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea3c136eb0 con 0x7fea3c1430c0 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 349989950 0 0) 0x7fea3c136230 con 0x7fea3c0a98f0 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea3c137190 con 0x7fea3c0a98f0 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fea38001a20 con 0x7fea3c0a98f0 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 10 monclient(hunting): dump: 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:35.580 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:35.581 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:35.581 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.581 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.578+0000 7fea327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:35.582 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.581+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1023872151 0 0) 0x7fea20004eb0 con 0x7fea3c13f710 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.581+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea3c136230 con 0x7fea3c13f710 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.581+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fea2c001930 con 0x7fea3c13f710 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.581+0000 7fea327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.581+0000 7fea327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.581+0000 7fea327fc640 10 monclient(hunting): dump: 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:35.583 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4246348555 0 0) 0x7fea3c136230 con 0x7fea3c13f710 2024-03-25T18:23:35.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 >> v1:172.21.15.132:6790/0 conn(0x7fea3c1430c0 legacy=0x7fea3c1db4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 >> v1:172.21.15.161:6789/0 conn(0x7fea3c0a98f0 legacy=0x7fea3c137890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:35.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 1 monclient: found mon.a 2024-03-25T18:23:35.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.591+0000 7fea327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c137470 con 0x7fea3c13f710 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea327fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 10 monclient: _renew_subs 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c0b7bb0 con 0x7fea3c13f710 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 10 monclient: _renew_subs 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c135170 con 0x7fea3c13f710 2024-03-25T18:23:35.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.592+0000 7fea43ad6640 1 librados: init done 2024-03-25T18:23:35.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.593+0000 7fea43ad6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.593+0000 7fea43ad6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.593+0000 7fea43ad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:35.595 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.593+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c0627b0 con 0x7fea3c13f710 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea2c003610 con 0x7fea3c13f710 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fea2c003920 con 0x7fea3c13f710 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 10 monclient: dump: 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:35.892 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:35.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea33fff640 4 set_mon_vals no callback set 2024-03-25T18:23:35.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.891+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fea2c019970 con 0x7fea3c13f710 2024-03-25T18:23:35.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.892+0000 7fea41872640 10 monclient: get_auth_request con 0x7fea2006dfb0 auth_method 0 2024-03-25T18:23:35.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.892+0000 7fea327fc640 10 client.4659.objecter ms_handle_connect 0x7fea2006dfb0 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1661381743 0 0) 0x7fea2c0535f0 con 0x7fea3c13f710 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 10 client.4659.objecter ms_dispatch 0x7fea3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea2c04ef30 con 0x7fea3c13f710 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.904 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:35.902+0000 7fea327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:36.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.120+0000 7fea43ad6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:36.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.120+0000 7fea43ad6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:36.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.120+0000 7fea43ad6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:36.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.120+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea3c13e100 con 0x7fea3c13f710 2024-03-25T18:23:36.138 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:21.402 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.187 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:21.450 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.218 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:21.482 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.234 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:21.475 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.245 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:21.486 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.251 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:21.515 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.263 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:21.527 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.266 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:21.508 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.272+0000 7fea327fc640 1 -- 172.21.15.132:0/2925260738 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7fea2c051030 con 0x7fea3c13f710 2024-03-25T18:23:36.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.272+0000 7fea327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:36.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.272+0000 7fea327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:36.274 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:36.274 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:36.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.275+0000 7fea43ad6640 10 librados: watch_flush enter 2024-03-25T18:23:36.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.275+0000 7fea43ad6640 10 librados: watch_flush exit 2024-03-25T18:23:36.277 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.275+0000 7fea43ad6640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.275+0000 7fea43ad6640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 >> v1:172.21.15.161:6800/411950978 conn(0x7fea2006dfb0 legacy=0x7fea20070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 10 monclient: shutdown 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 >> v1:172.21.15.132:6789/0 conn(0x7fea3c13f710 legacy=0x7fea3c0b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 shutdown_connections 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 >> 172.21.15.132:0/2925260738 conn(0x7fea3c0b7ea0 msgr2=0x7fea3c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 shutdown_connections 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 -- 172.21.15.132:0/2925260738 wait complete. 2024-03-25T18:23:36.278 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:36.276+0000 7fea43ad6640 1 librados: shutdown 2024-03-25T18:23:36.288 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:21.552 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:36.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:43.096 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.201+0000 7fcedccd3640 10 monclient: get_monmap_and_config 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.201+0000 7fcedccd3640 10 monclient: build_initial_monmap 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.201+0000 7fcedccd3640 10 monclient: monmap: 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:43.202929+0000 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:43.202929+0000 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:43.203 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:43.204 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.202+0000 7fcedccd3640 1 Processor -- start 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.202+0000 7fcedccd3640 1 -- start start 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.202+0000 7fcedccd3640 10 monclient: init 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient(hunting): picked mon.noname-c con 0x7fced80b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient(hunting): picked mon.noname-b con 0x7fced80b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient(hunting): picked mon.noname-a con 0x7fced814be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80af940 con 0x7fced814be90 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80b0a70 con 0x7fced80b4d60 2024-03-25T18:23:43.205 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced80b0d50 con 0x7fced80b1730 2024-03-25T18:23:43.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:43.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fcedccd3640 10 monclient(hunting): authenticate will time out at 7863.465939s 2024-03-25T18:23:43.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fced659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fced80b4d60 0x7fced813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:60360/0 (socket says 172.21.15.132:60360) 2024-03-25T18:23:43.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.203+0000 7fced659c640 1 -- 172.21.15.132:0/2840485922 learned_addr learned my addr 172.21.15.132:0/2840485922 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:43.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.272+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502577449 0 0) 0x7fced80b0d50 con 0x7fced80b1730 2024-03-25T18:23:43.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.272+0000 7fced5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.272+0000 7fced5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.272+0000 7fced5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.272+0000 7fced5d9b640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:23:43.274 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.272+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcec4003520 con 0x7fced80b1730 2024-03-25T18:23:43.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.289+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154992640 0 0) 0x7fced80b0a70 con 0x7fced80b4d60 2024-03-25T18:23:43.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.289+0000 7fced5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.289+0000 7fced5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.289+0000 7fced5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.289+0000 7fced5d9b640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:23:43.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.289+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced80b0d50 con 0x7fced80b4d60 2024-03-25T18:23:43.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.294+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106736455 0 0) 0x7fced80af940 con 0x7fced814be90 2024-03-25T18:23:43.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.294+0000 7fced5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.294+0000 7fced5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.294+0000 7fced5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.294+0000 7fced5d9b640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:23:43.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.294+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced80b0a70 con 0x7fced814be90 2024-03-25T18:23:43.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4075604478 0 0) 0x7fcec4003520 con 0x7fced80b1730 2024-03-25T18:23:43.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 >> v1:172.21.15.161:6789/0 conn(0x7fced80b4d60 legacy=0x7fced813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 >> v1:172.21.15.132:6789/0 conn(0x7fced814be90 legacy=0x7fced814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:23:43.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 1 monclient: found mon.noname-b 2024-03-25T18:23:43.300 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80b1030 con 0x7fced80b1730 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fcec0001690 con 0x7fced80b1730 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.298+0000 7fced5d9b640 10 monclient: dump: 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:43.301 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:43.302 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:43.302 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:43.302 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:43.302 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.299+0000 7fcedccd3640 5 monclient: authenticate success, global_id 4676 2024-03-25T18:23:43.302 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.299+0000 7fcedccd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec00017f0 con 0x7fced80b1730 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fced5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fced5d9b640 1 -- 172.21.15.132:0/2840485922 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fcec0001c00 con 0x7fced80b1730 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fced5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fced5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fced5d9b640 10 monclient: dump: 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:43.318 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fcedccd3640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:43.319 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fcedccd3640 4 set_mon_vals no callback set 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fcedccd3640 10 monclient: shutdown 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fcedccd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.317+0000 7fcedccd3640 1 -- 172.21.15.132:0/2840485922 >> v1:172.21.15.132:6790/0 conn(0x7fced80b1730 legacy=0x7fced80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.318+0000 7fcedccd3640 1 -- 172.21.15.132:0/2840485922 shutdown_connections 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.318+0000 7fcedccd3640 1 -- 172.21.15.132:0/2840485922 >> 172.21.15.132:0/2840485922 conn(0x7fced80617a0 msgr2=0x7fced8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.318+0000 7fcedccd3640 1 -- 172.21.15.132:0/2840485922 shutdown_connections 2024-03-25T18:23:43.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.318+0000 7fcedccd3640 1 -- 172.21.15.132:0/2840485922 wait complete. 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.319+0000 7fcedccd3640 10 monclient: build_initial_monmap 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.319+0000 7fcedccd3640 10 monclient: monmap: 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:43.321 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.319+0000 7fcedccd3640 1 librados: starting msgr at 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.319+0000 7fcedccd3640 1 librados: starting objecter 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.319+0000 7fcedccd3640 1 Processor -- start 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 1 -- start start 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 1 librados: setting wanted keys 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 1 librados: calling monclient init 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient: init 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient(hunting): picked mon.noname-b con 0x7fced80b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:43.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient(hunting): picked mon.noname-a con 0x7fced80b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:43.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient(hunting): picked mon.noname-c con 0x7fced814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:43.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced81d7980 con 0x7fced80b4d60 2024-03-25T18:23:43.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced81420a0 con 0x7fced80b1730 2024-03-25T18:23:43.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced81432c0 con 0x7fced814be90 2024-03-25T18:23:43.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:43.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.320+0000 7fcedccd3640 10 monclient(hunting): authenticate will time out at 7863.583675s 2024-03-25T18:23:43.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.327+0000 7fcecffff640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fced80b4d60 0x7fced8138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:33376/0 (socket says 172.21.15.132:33376) 2024-03-25T18:23:43.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.327+0000 7fcecffff640 1 -- 172.21.15.132:0/1657976301 learned_addr learned my addr 172.21.15.132:0/1657976301 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:43.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.331+0000 7fceceffd640 10 client.?.objecter ms_handle_connect 0x7fced80b4d60 2024-03-25T18:23:43.332 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.331+0000 7fceceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:43.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.332+0000 7fceceffd640 10 client.?.objecter ms_handle_connect 0x7fced80b1730 2024-03-25T18:23:43.333 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.332+0000 7fceceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.337+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033411339 0 0) 0x7fced81d7980 con 0x7fced80b4d60 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.337+0000 7fceceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.337+0000 7fceceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.337+0000 7fceceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.337+0000 7fceceffd640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.337+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb8004eb0 con 0x7fced80b4d60 2024-03-25T18:23:43.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.347+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521939985 0 0) 0x7fced81420a0 con 0x7fced80b1730 2024-03-25T18:23:43.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.347+0000 7fceceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.347+0000 7fceceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.347+0000 7fceceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.347+0000 7fceceffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:23:43.348 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.347+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced81d7980 con 0x7fced80b1730 2024-03-25T18:23:43.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.348+0000 7fceceffd640 10 client.?.objecter ms_handle_connect 0x7fced814be90 2024-03-25T18:23:43.349 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.348+0000 7fceceffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:43.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4260043861 0 0) 0x7fceb8004eb0 con 0x7fced80b4d60 2024-03-25T18:23:43.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fced81420a0 con 0x7fced80b4d60 2024-03-25T18:23:43.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fcec0001bc0 con 0x7fced80b4d60 2024-03-25T18:23:43.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.352 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 10 monclient(hunting): dump: 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:43.353 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.351+0000 7fceceffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:43.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.357+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303529745 0 0) 0x7fced81432c0 con 0x7fced814be90 2024-03-25T18:23:43.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.357+0000 7fceceffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:43.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.357+0000 7fceceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:43.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.357+0000 7fceceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:43.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.357+0000 7fceceffd640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:23:43.358 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.357+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceb8004eb0 con 0x7fced814be90 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1538444801 0 0) 0x7fced81420a0 con 0x7fced80b4d60 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 >> v1:172.21.15.132:6790/0 conn(0x7fced814be90 legacy=0x7fced81d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 >> v1:172.21.15.161:6789/0 conn(0x7fced80b1730 legacy=0x7fced8141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 1 monclient: found mon.a 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8143940 con 0x7fced80b4d60 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fceceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 5 monclient: authenticate success, global_id 4665 2024-03-25T18:23:43.360 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 10 monclient: _renew_subs 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced8138a70 con 0x7fced80b4d60 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 10 monclient: _renew_subs 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fced81d8700 con 0x7fced80b4d60 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.358+0000 7fcedccd3640 1 librados: init done 2024-03-25T18:23:43.361 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.359+0000 7fcedccd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.359+0000 7fcedccd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.359+0000 7fcedccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:43.362 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.359+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced8143130 con 0x7fced80b4d60 2024-03-25T18:23:43.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcec00011f0 con 0x7fced80b4d60 2024-03-25T18:23:43.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:43.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fcec0003b00 con 0x7fced80b4d60 2024-03-25T18:23:43.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 10 monclient: dump: 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:43.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fced559a640 4 set_mon_vals no callback set 2024-03-25T18:23:43.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.669+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fcec0003d20 con 0x7fced80b4d60 2024-03-25T18:23:43.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.670+0000 7fced659c640 10 monclient: get_auth_request con 0x7fceb806de00 auth_method 0 2024-03-25T18:23:43.672 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.670+0000 7fceceffd640 10 client.4665.objecter ms_handle_connect 0x7fceb806de00 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7fcec0054880 con 0x7fced80b4d60 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 10 client.4665.objecter ms_dispatch 0x7fced8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 3 client.4665.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 3 client.4665.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcec0050390 con 0x7fced80b4d60 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.678 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.676+0000 7fceceffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:43.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.916+0000 7fcedccd3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.916+0000 7fcedccd3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.916+0000 7fcedccd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:43.917 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.916+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fced81401f0 con 0x7fced80b4d60 2024-03-25T18:23:43.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.949+0000 7fceceffd640 1 -- 172.21.15.132:0/1657976301 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7fcec00522c0 con 0x7fced80b4d60 2024-03-25T18:23:43.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.949+0000 7fceceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.950 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.949+0000 7fceceffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:43.950 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:43.951 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:43.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.951+0000 7fcedccd3640 10 librados: watch_flush enter 2024-03-25T18:23:43.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.951+0000 7fcedccd3640 10 librados: watch_flush exit 2024-03-25T18:23:43.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.951+0000 7fcedccd3640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.951+0000 7fcedccd3640 10 client.4665.objecter successfully canceled tick 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.951+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 >> v1:172.21.15.161:6800/411950978 conn(0x7fceb806de00 legacy=0x7fceb80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 10 monclient: shutdown 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 >> v1:172.21.15.132:6789/0 conn(0x7fced80b4d60 legacy=0x7fced8138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 shutdown_connections 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 >> 172.21.15.132:0/1657976301 conn(0x7fced80617a0 msgr2=0x7fced8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 shutdown_connections 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 1 -- 172.21.15.132:0/1657976301 wait complete. 2024-03-25T18:23:43.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:43.952+0000 7fcedccd3640 1 librados: shutdown 2024-03-25T18:23:43.966 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:50.773 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:50.884 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.883+0000 7f073cce4640 10 monclient: get_monmap_and_config 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.883+0000 7f073cce4640 10 monclient: build_initial_monmap 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.883+0000 7f073cce4640 10 monclient: monmap: 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:50.884951+0000 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:50.884951+0000 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:50.885 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:50.886 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:50.886 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.884+0000 7f073cce4640 1 Processor -- start 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.884+0000 7f073cce4640 1 -- start start 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.884+0000 7f073cce4640 10 monclient: init 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient(hunting): picked mon.noname-c con 0x7f07380b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient(hunting): picked mon.noname-a con 0x7f07380b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient(hunting): picked mon.noname-b con 0x7f073814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07380af940 con 0x7f07380b1730 2024-03-25T18:23:50.887 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07380b0a70 con 0x7f07380b4d60 2024-03-25T18:23:50.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07380b0d50 con 0x7f073814be90 2024-03-25T18:23:50.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:50.888 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.885+0000 7f073cce4640 10 monclient(hunting): authenticate will time out at 7871.148173s 2024-03-25T18:23:50.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.892+0000 7f073659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f07380b4d60 0x7f073813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:36704/0 (socket says 172.21.15.132:36704) 2024-03-25T18:23:50.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.892+0000 7f073659c640 1 -- 172.21.15.132:0/2875108672 learned_addr learned my addr 172.21.15.132:0/2875108672 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:50.910 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.909+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017261409 0 0) 0x7f07380b0a70 con 0x7f07380b4d60 2024-03-25T18:23:50.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.909+0000 7f0735d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.909+0000 7f0735d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.909+0000 7f0735d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.909+0000 7f0735d9b640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:23:50.911 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.909+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0724003520 con 0x7f07380b4d60 2024-03-25T18:23:50.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 44094538 0 0) 0x7f0724003520 con 0x7f07380b4d60 2024-03-25T18:23:50.921 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 >> v1:172.21.15.132:6790/0 conn(0x7f073814be90 legacy=0x7f073814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 >> v1:172.21.15.132:6789/0 conn(0x7f07380b1730 legacy=0x7f07380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 1 monclient: found mon.noname-c 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07380b1030 con 0x7f07380b4d60 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: _finish_auth 0 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0728001690 con 0x7f07380b4d60 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:50.922 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f0735d9b640 10 monclient: dump: 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f073cce4640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:23:50.923 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.920+0000 7f073cce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07280017f0 con 0x7f07380b4d60 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f0735d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f0735d9b640 1 -- 172.21.15.132:0/2875108672 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0728001c00 con 0x7f07380b4d60 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f073cce4640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f073cce4640 4 set_mon_vals no callback set 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f0735d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f0735d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:50.936 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f0735d9b640 10 monclient: dump: 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:50.937 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f073cce4640 10 monclient: shutdown 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f073cce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.935+0000 7f073cce4640 1 -- 172.21.15.132:0/2875108672 >> v1:172.21.15.161:6789/0 conn(0x7f07380b4d60 legacy=0x7f073813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.936+0000 7f073cce4640 1 -- 172.21.15.132:0/2875108672 shutdown_connections 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.936+0000 7f073cce4640 1 -- 172.21.15.132:0/2875108672 >> 172.21.15.132:0/2875108672 conn(0x7f07380617a0 msgr2=0x7f0738061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.936+0000 7f073cce4640 1 -- 172.21.15.132:0/2875108672 shutdown_connections 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.936+0000 7f073cce4640 1 -- 172.21.15.132:0/2875108672 wait complete. 2024-03-25T18:23:50.940 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 10 monclient: build_initial_monmap 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 10 monclient: monmap: 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 1 librados: starting msgr at 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 1 librados: starting objecter 2024-03-25T18:23:50.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 1 Processor -- start 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 1 -- start start 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 1 librados: setting wanted keys 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 1 librados: calling monclient init 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.937+0000 7f073cce4640 10 monclient: init 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient(hunting): picked mon.noname-c con 0x7f07380b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient(hunting): picked mon.noname-b con 0x7f07380b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient(hunting): picked mon.noname-a con 0x7f073814be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07381d7940 con 0x7f073814be90 2024-03-25T18:23:50.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07381382f0 con 0x7f07380b4d60 2024-03-25T18:23:50.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0738137320 con 0x7f07380b1730 2024-03-25T18:23:50.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:50.943 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.938+0000 7f073cce4640 10 monclient(hunting): authenticate will time out at 7871.201130s 2024-03-25T18:23:50.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.946+0000 7f072ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f07380b4d60 0x7f07381d7e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:36754/0 (socket says 172.21.15.132:36754) 2024-03-25T18:23:50.947 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.946+0000 7f072ffff640 1 -- 172.21.15.132:0/1972812006 learned_addr learned my addr 172.21.15.132:0/1972812006 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:50.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.947+0000 7f072effd640 10 client.?.objecter ms_handle_connect 0x7f07380b4d60 2024-03-25T18:23:50.949 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.947+0000 7f072effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.953+0000 7f072effd640 10 client.?.objecter ms_handle_connect 0x7f073814be90 2024-03-25T18:23:50.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.953+0000 7f072effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 10 client.?.objecter ms_handle_connect 0x7f07380b1730 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230057121 0 0) 0x7f07381382f0 con 0x7f07380b4d60 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.962+0000 7f072effd640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:23:50.964 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.963+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0718004eb0 con 0x7f07380b4d60 2024-03-25T18:23:50.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.968+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962478511 0 0) 0x7f07381d7940 con 0x7f073814be90 2024-03-25T18:23:50.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.969+0000 7f072effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.969+0000 7f072effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.969+0000 7f072effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.969+0000 7f072effd640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:23:50.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.969+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07381382f0 con 0x7f073814be90 2024-03-25T18:23:50.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263348975 0 0) 0x7f0718004eb0 con 0x7f07380b4d60 2024-03-25T18:23:50.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07381d7940 con 0x7f07380b4d60 2024-03-25T18:23:50.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0720001a30 con 0x7f07380b4d60 2024-03-25T18:23:50.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.986 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 10 monclient(hunting): dump: 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:50.987 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.985+0000 7f072effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:50.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.988+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773947555 0 0) 0x7f0738137320 con 0x7f07380b1730 2024-03-25T18:23:50.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.988+0000 7f072effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.988+0000 7f072effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.988+0000 7f072effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.988+0000 7f072effd640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:23:50.989 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.988+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0718004eb0 con 0x7f07380b1730 2024-03-25T18:23:50.993 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.992+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1657876929 0 0) 0x7f07381382f0 con 0x7f073814be90 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.992+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0738137320 con 0x7f073814be90 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.992+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0730001b00 con 0x7f073814be90 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.992+0000 7f072effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.992+0000 7f072effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.992+0000 7f072effd640 10 monclient(hunting): dump: 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:50.994 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:50.995 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:50.995 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:50.995 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2726238381 0 0) 0x7f07381d7940 con 0x7f07380b4d60 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 >> v1:172.21.15.132:6790/0 conn(0x7f07380b1730 legacy=0x7f0738136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 >> v1:172.21.15.132:6789/0 conn(0x7f073814be90 legacy=0x7f073812fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 1 monclient: found mon.b 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07381d8530 con 0x7f07380b4d60 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 10 monclient: _finish_auth 0 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f072effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 10 monclient: _renew_subs 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:51.000 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0738130570 con 0x7f07380b4d60 2024-03-25T18:23:51.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:51.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 10 monclient: _renew_subs 2024-03-25T18:23:51.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:51.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0738143ae0 con 0x7f07380b4d60 2024-03-25T18:23:51.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.997+0000 7f073cce4640 1 librados: init done 2024-03-25T18:23:51.001 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.999+0000 7f073cce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:51.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.999+0000 7f073cce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:51.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.999+0000 7f073cce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:51.002 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:50.999+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07381d77a0 con 0x7f07380b4d60 2024-03-25T18:23:51.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0720001c70 con 0x7f07380b4d60 2024-03-25T18:23:51.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f072effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f07200038c0 con 0x7f07380b4d60 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f072effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f072effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f072effd640 10 monclient: dump: 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:51.282 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:51.283 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:51.283 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:51.283 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:51.283 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:51.283 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.280+0000 7f073559a640 4 set_mon_vals no callback set 2024-03-25T18:23:51.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.281+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f0720003aa0 con 0x7f07380b4d60 2024-03-25T18:23:51.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.281+0000 7f073659c640 10 monclient: get_auth_request con 0x7f071806dfb0 auth_method 0 2024-03-25T18:23:51.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.282+0000 7f072effd640 10 client.4693.objecter ms_handle_connect 0x7f071806dfb0 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.286+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f0720054710 con 0x7f07380b4d60 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.286+0000 7f072effd640 10 client.4693.objecter ms_dispatch 0x7f0738000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.286+0000 7f072effd640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.286+0000 7f072effd640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.286+0000 7f072effd640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.287+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0720050220 con 0x7f07380b4d60 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.287+0000 7f072effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:51.288 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.287+0000 7f072effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:51.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.516+0000 7f073cce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.516+0000 7f073cce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.516+0000 7f073cce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:23:51.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.516+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07381d7b40 con 0x7f07380b4d60 2024-03-25T18:23:51.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.547+0000 7f072effd640 1 -- 172.21.15.132:0/1972812006 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f07381d7b40 con 0x7f07380b4d60 2024-03-25T18:23:51.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.547+0000 7f072effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.548 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.547+0000 7f072effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:51.549 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:51.549 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:51.551 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.549+0000 7f073cce4640 10 librados: watch_flush enter 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 10 librados: watch_flush exit 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 >> v1:172.21.15.161:6800/411950978 conn(0x7f071806dfb0 legacy=0x7f0718070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 10 monclient: shutdown 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 >> v1:172.21.15.161:6789/0 conn(0x7f07380b4d60 legacy=0x7f07381d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 shutdown_connections 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 >> 172.21.15.132:0/1972812006 conn(0x7f07380617a0 msgr2=0x7f07380b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 shutdown_connections 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 -- 172.21.15.132:0/1972812006 wait complete. 2024-03-25T18:23:51.552 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:51.550+0000 7f073cce4640 1 librados: shutdown 2024-03-25T18:23:51.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:55.972 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:41.236 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:55.979 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:41.243 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.024 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:10:41.288 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.032 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:41.274 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.036 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:41.300 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.037 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:41.278 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.056 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:41.320 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.063 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:10:41.304 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:56.093 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:10:41.357 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:58.369 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.471+0000 7fe9c4ea8640 10 monclient: get_monmap_and_config 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.471+0000 7fe9c4ea8640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.471+0000 7fe9c4ea8640 10 monclient: monmap: 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:23:58.472939+0000 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:23:58.472939+0000 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:58.473 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:23:58.474 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:23:58.474 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.472+0000 7fe9c4ea8640 1 Processor -- start 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.472+0000 7fe9c4ea8640 1 -- start start 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.472+0000 7fe9c4ea8640 10 monclient: init 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c00a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c013f710 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c01430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0142520 con 0x7fe9c01430c0 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00b28a0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c00b2a30 con 0x7fe9c013f710 2024-03-25T18:23:58.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.473+0000 7fe9c4ea8640 10 monclient(hunting): authenticate will time out at 7878.736113s 2024-03-25T18:23:58.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.485+0000 7fe9be59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe9c00a98f0 0x7fe9c00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:55176/0 (socket says 172.21.15.132:55176) 2024-03-25T18:23:58.486 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.485+0000 7fe9be59c640 1 -- 172.21.15.132:0/3153957797 learned_addr learned my addr 172.21.15.132:0/3153957797 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:58.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.501+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661112563 0 0) 0x7fe9c00b28a0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.501+0000 7fe9bd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.501+0000 7fe9bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.501+0000 7fe9bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.502 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.501+0000 7fe9bd59a640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:23:58.503 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.501+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a8003520 con 0x7fe9c00a98f0 2024-03-25T18:23:58.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.508+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021677948 0 0) 0x7fe9c00b2a30 con 0x7fe9c013f710 2024-03-25T18:23:58.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.508+0000 7fe9bd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.508+0000 7fe9bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.508+0000 7fe9bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.508+0000 7fe9bd59a640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:23:58.509 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.508+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c00b28a0 con 0x7fe9c013f710 2024-03-25T18:23:58.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.515+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125180300 0 0) 0x7fe9c0142520 con 0x7fe9c01430c0 2024-03-25T18:23:58.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.515+0000 7fe9bd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.515+0000 7fe9bd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.515+0000 7fe9bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.516 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.515+0000 7fe9bd59a640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:23:58.517 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.515+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c00b2a30 con 0x7fe9c01430c0 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728615852 0 0) 0x7fe9c00b28a0 con 0x7fe9c013f710 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 >> v1:172.21.15.161:6789/0 conn(0x7fe9c00a98f0 legacy=0x7fe9c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 >> v1:172.21.15.132:6789/0 conn(0x7fe9c01430c0 legacy=0x7fe9c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 1 monclient: found mon.noname-b 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c00b44c0 con 0x7fe9c013f710 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: _finish_auth 0 2024-03-25T18:23:58.518 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe9b4001670 con 0x7fe9c013f710 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9bd59a640 10 monclient: dump: 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.519 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9c4ea8640 5 monclient: authenticate success, global_id 4688 2024-03-25T18:23:58.520 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.516+0000 7fe9c4ea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b40017d0 con 0x7fe9c013f710 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9bd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9bd59a640 1 -- 172.21.15.132:0/3153957797 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe9b4001be0 con 0x7fe9c013f710 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9c4ea8640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9c4ea8640 4 set_mon_vals no callback set 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9bd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9bd59a640 10 monclient: dump: 2024-03-25T18:23:58.524 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:58.525 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.526 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9c4ea8640 10 monclient: shutdown 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9c4ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.523+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3153957797 >> v1:172.21.15.132:6790/0 conn(0x7fe9c013f710 legacy=0x7fe9c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.524+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3153957797 shutdown_connections 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.524+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3153957797 >> 172.21.15.132:0/3153957797 conn(0x7fe9c00b7ea0 msgr2=0x7fe9c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.524+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3153957797 shutdown_connections 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.524+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3153957797 wait complete. 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 10 monclient: monmap: 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:23:58.527 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 1 librados: starting msgr at 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 1 librados: starting objecter 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 1 Processor -- start 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 1 -- start start 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 1 librados: setting wanted keys 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 1 librados: calling monclient init 2024-03-25T18:23:58.528 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.525+0000 7fe9c4ea8640 10 monclient: init 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c00a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c013f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c01430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c0136e60 con 0x7fe9c013f710 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c01381a0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c01370b0 con 0x7fe9c01430c0 2024-03-25T18:23:58.529 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.530 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.526+0000 7fe9c4ea8640 10 monclient(hunting): authenticate will time out at 7878.789164s 2024-03-25T18:23:58.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.536+0000 7fe9bdd9b640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fe9c013f710 0x7fe9c013a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:37802/0 (socket says 172.21.15.132:37802) 2024-03-25T18:23:58.537 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.536+0000 7fe9bdd9b640 1 -- 172.21.15.132:0/3353986366 learned_addr learned my addr 172.21.15.132:0/3353986366 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:23:58.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.537+0000 7fe9aeffd640 10 client.?.objecter ms_handle_connect 0x7fe9c013f710 2024-03-25T18:23:58.538 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.537+0000 7fe9aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.546+0000 7fe9aeffd640 10 client.?.objecter ms_handle_connect 0x7fe9c00a98f0 2024-03-25T18:23:58.547 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.546+0000 7fe9aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.554 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.553+0000 7fe9aeffd640 10 client.?.objecter ms_handle_connect 0x7fe9c01430c0 2024-03-25T18:23:58.554 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.553+0000 7fe9aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.559+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623197548 0 0) 0x7fe9c0136e60 con 0x7fe9c013f710 2024-03-25T18:23:58.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.559+0000 7fe9aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.559+0000 7fe9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.559+0000 7fe9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.559+0000 7fe9aeffd640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:23:58.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.559+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a0004eb0 con 0x7fe9c013f710 2024-03-25T18:23:58.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.562+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 401864906 0 0) 0x7fe9c01370b0 con 0x7fe9c01430c0 2024-03-25T18:23:58.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.562+0000 7fe9aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.562+0000 7fe9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.563 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.562+0000 7fe9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.562+0000 7fe9aeffd640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:23:58.564 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.562+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0136e60 con 0x7fe9c01430c0 2024-03-25T18:23:58.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.567+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160497868 0 0) 0x7fe9c01381a0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.567+0000 7fe9aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.567+0000 7fe9aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.567+0000 7fe9aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.567+0000 7fe9aeffd640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:23:58.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.567+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c01370b0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 591164560 0 0) 0x7fe9a0004eb0 con 0x7fe9c013f710 2024-03-25T18:23:58.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c01381a0 con 0x7fe9c013f710 2024-03-25T18:23:58.571 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe9b4001670 con 0x7fe9c013f710 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 10 monclient(hunting): dump: 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.572 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.570+0000 7fe9aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.586+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2430774599 0 0) 0x7fe9c01370b0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.586+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a0004eb0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.586+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe9b00017b0 con 0x7fe9c00a98f0 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.586+0000 7fe9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.586+0000 7fe9aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.586+0000 7fe9aeffd640 10 monclient(hunting): dump: 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.587 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.588 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.588 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.588 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:58.588 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:58.588 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:58.588 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.591+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860661243 0 0) 0x7fe9c0136e60 con 0x7fe9c01430c0 2024-03-25T18:23:58.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.591+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c01370b0 con 0x7fe9c01430c0 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.591+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe9b8001b50 con 0x7fe9c01430c0 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.591+0000 7fe9aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.591+0000 7fe9aeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.591+0000 7fe9aeffd640 10 monclient(hunting): dump: 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:58.593 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2994725619 0 0) 0x7fe9c01381a0 con 0x7fe9c013f710 2024-03-25T18:23:58.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 >> v1:172.21.15.132:6790/0 conn(0x7fe9c01430c0 legacy=0x7fe9c013ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 >> v1:172.21.15.161:6789/0 conn(0x7fe9c00a98f0 legacy=0x7fe9c0137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:58.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 1 monclient: found mon.a 2024-03-25T18:23:58.602 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0137390 con 0x7fe9c013f710 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9c4ea8640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9c4ea8640 10 monclient: _renew_subs 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9c4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c01376d0 con 0x7fe9c013f710 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9c4ea8640 10 client.4677.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.600+0000 7fe9c4ea8640 10 monclient: _renew_subs 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.601+0000 7fe9c4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.601+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c0135530 con 0x7fe9c013f710 2024-03-25T18:23:58.603 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.601+0000 7fe9c4ea8640 1 librados: init done 2024-03-25T18:23:58.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.602+0000 7fe9c4ea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.602+0000 7fe9c4ea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.602+0000 7fe9c4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:58.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.602+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c00b3ef0 con 0x7fe9c013f710 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9b40039e0 con 0x7fe9c013f710 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fe9b4003e30 con 0x7fe9c013f710 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9aeffd640 10 monclient: dump: 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:23:58.930 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.931 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:23:58.931 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:23:58.931 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:23:58.931 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:23:58.931 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:23:58.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.928+0000 7fe9bcd99640 4 set_mon_vals no callback set 2024-03-25T18:23:58.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.929+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fe9b4004050 con 0x7fe9c013f710 2024-03-25T18:23:58.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.930+0000 7fe9be59c640 10 monclient: get_auth_request con 0x7fe9a006e0f0 auth_method 0 2024-03-25T18:23:58.932 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.930+0000 7fe9aeffd640 10 client.4677.objecter ms_handle_connect 0x7fe9a006e0f0 2024-03-25T18:23:58.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7fe9b4054930 con 0x7fe9c013f710 2024-03-25T18:23:58.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 10 client.4677.objecter ms_dispatch 0x7fe9c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:23:58.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 3 client.4677.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:23:58.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 3 client.4677.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:23:58.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 20 client.4677.objecter dump_active .. 0 homeless 2024-03-25T18:23:58.937 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9b4050440 con 0x7fe9c013f710 2024-03-25T18:23:58.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.938 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:58.936+0000 7fe9aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:59.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.181+0000 7fe9c4ea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:59.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.181+0000 7fe9c4ea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:59.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.181+0000 7fe9c4ea8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:23:59.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.181+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c0137520 con 0x7fe9c013f710 2024-03-25T18:23:59.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.206+0000 7fe9aeffd640 1 -- 172.21.15.132:0/3353986366 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7fe9b4052370 con 0x7fe9c013f710 2024-03-25T18:23:59.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.206+0000 7fe9aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:59.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.206+0000 7fe9aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:59.208 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:23:59.208 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 10 librados: watch_flush enter 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 10 librados: watch_flush exit 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 20 client.4677.objecter shutdown clearing up homeless session... 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 10 client.4677.objecter successfully canceled tick 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 >> v1:172.21.15.161:6800/411950978 conn(0x7fe9a006e0f0 legacy=0x7fe9a00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 10 monclient: shutdown 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:59.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 >> v1:172.21.15.132:6789/0 conn(0x7fe9c013f710 legacy=0x7fe9c013a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 shutdown_connections 2024-03-25T18:23:59.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 >> 172.21.15.132:0/3353986366 conn(0x7fe9c00b7ea0 msgr2=0x7fe9c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:59.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 shutdown_connections 2024-03-25T18:23:59.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 -- 172.21.15.132:0/3353986366 wait complete. 2024-03-25T18:23:59.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:23:59.209+0000 7fe9c4ea8640 1 librados: shutdown 2024-03-25T18:23:59.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:06.030 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.146+0000 7fee83ffb640 10 monclient: get_monmap_and_config 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.146+0000 7fee83ffb640 10 monclient: build_initial_monmap 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.146+0000 7fee83ffb640 10 monclient: monmap: 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:06.147866+0000 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:06.147866+0000 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:06.148 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:06.149 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.147+0000 7fee83ffb640 1 Processor -- start 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.147+0000 7fee83ffb640 1 -- start start 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.147+0000 7fee83ffb640 10 monclient: init 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fee7c0a98f0 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fee7c13f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fee7c1430c0 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c142520 con 0x7fee7c1430c0 2024-03-25T18:24:06.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c0b28a0 con 0x7fee7c13f710 2024-03-25T18:24:06.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c0b2a30 con 0x7fee7c0a98f0 2024-03-25T18:24:06.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:06.151 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.148+0000 7fee83ffb640 10 monclient(hunting): authenticate will time out at 7886.411542s 2024-03-25T18:24:06.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.157+0000 7fee81596640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fee7c13f710 0x7fee7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:60428/0 (socket says 172.21.15.132:60428) 2024-03-25T18:24:06.158 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.157+0000 7fee81596640 1 -- 172.21.15.132:0/3278130332 learned_addr learned my addr 172.21.15.132:0/3278130332 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:06.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.169+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192474937 0 0) 0x7fee7c0b28a0 con 0x7fee7c13f710 2024-03-25T18:24:06.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.169+0000 7fee80d95640 10 monclient(hunting): _negotiate 2024-03-25T18:24:06.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.169+0000 7fee80d95640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:06.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.169+0000 7fee80d95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:06.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.169+0000 7fee80d95640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:24:06.170 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.169+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee68003520 con 0x7fee7c13f710 2024-03-25T18:24:06.189 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.187+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377861589 0 0) 0x7fee7c0b2a30 con 0x7fee7c0a98f0 2024-03-25T18:24:06.189 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.187+0000 7fee80d95640 10 monclient(hunting): _negotiate 2024-03-25T18:24:06.189 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.187+0000 7fee80d95640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:06.189 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.187+0000 7fee80d95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:06.189 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.187+0000 7fee80d95640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:24:06.189 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.187+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee7c0b28a0 con 0x7fee7c0a98f0 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2445332967 0 0) 0x7fee68003520 con 0x7fee7c13f710 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 >> v1:172.21.15.132:6790/0 conn(0x7fee7c0a98f0 legacy=0x7fee7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 >> v1:172.21.15.132:6789/0 conn(0x7fee7c1430c0 legacy=0x7fee7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: _finish_hunting 0 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 1 monclient: found mon.noname-c 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:24:06.197 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c0b44c0 con 0x7fee7c13f710 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: _finish_auth 0 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: _check_auth_tickets 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee6c001690 con 0x7fee7c13f710 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee80d95640 10 monclient: dump: 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:06.198 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:06.199 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:06.199 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:06.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee83ffb640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:24:06.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.195+0000 7fee83ffb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:06.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee6c0017f0 con 0x7fee7c13f710 2024-03-25T18:24:06.206 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee80d95640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee80d95640 1 -- 172.21.15.132:0/3278130332 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee6c001c00 con 0x7fee7c13f710 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee80d95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee80d95640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee80d95640 10 monclient: dump: 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:06.207 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:06.208 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:06.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee83ffb640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:06.208 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.205+0000 7fee83ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 10 monclient: shutdown 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 1 -- 172.21.15.132:0/3278130332 >> v1:172.21.15.161:6789/0 conn(0x7fee7c13f710 legacy=0x7fee7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 1 -- 172.21.15.132:0/3278130332 shutdown_connections 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 1 -- 172.21.15.132:0/3278130332 >> 172.21.15.132:0/3278130332 conn(0x7fee7c0b7ea0 msgr2=0x7fee7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 1 -- 172.21.15.132:0/3278130332 shutdown_connections 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.206+0000 7fee83ffb640 1 -- 172.21.15.132:0/3278130332 wait complete. 2024-03-25T18:24:06.209 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.207+0000 7fee83ffb640 10 monclient: build_initial_monmap 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.207+0000 7fee83ffb640 10 monclient: monmap: 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 1 librados: starting msgr at 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 1 librados: starting objecter 2024-03-25T18:24:06.210 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 1 Processor -- start 2024-03-25T18:24:06.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 1 -- start start 2024-03-25T18:24:06.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 1 librados: setting wanted keys 2024-03-25T18:24:06.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 1 librados: calling monclient init 2024-03-25T18:24:06.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.208+0000 7fee83ffb640 10 monclient: init 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient(hunting): picked mon.noname-b con 0x7fee7c0a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fee7c13f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient(hunting): picked mon.noname-c con 0x7fee7c1430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c1d77a0 con 0x7fee7c13f710 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c138e40 con 0x7fee7c0a98f0 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee7c0aba30 con 0x7fee7c1430c0 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee83ffb640 10 monclient(hunting): authenticate will time out at 7886.471884s 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee81596640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fee7c13f710 0x7fee7c1d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:41866/0 (socket says 172.21.15.132:41866) 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.209+0000 7fee81596640 1 -- 172.21.15.132:0/2474154609 learned_addr learned my addr 172.21.15.132:0/2474154609 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:06.212 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.210+0000 7fee727fc640 10 client.?.objecter ms_handle_connect 0x7fee7c13f710 2024-03-25T18:24:06.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.210+0000 7fee727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:06.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.212+0000 7fee727fc640 10 client.?.objecter ms_handle_connect 0x7fee7c0a98f0 2024-03-25T18:24:06.213 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.212+0000 7fee727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:06.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.275+0000 7fee727fc640 10 client.?.objecter ms_handle_connect 0x7fee7c1430c0 2024-03-25T18:24:06.276 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.275+0000 7fee727fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:06.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.284+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942250504 0 0) 0x7fee7c0aba30 con 0x7fee7c1430c0 2024-03-25T18:24:06.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.284+0000 7fee727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:06.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.284+0000 7fee727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:06.285 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.284+0000 7fee727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:06.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.284+0000 7fee727fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:24:06.286 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.284+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee60004eb0 con 0x7fee7c1430c0 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4290632891 0 0) 0x7fee60004eb0 con 0x7fee7c1430c0 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee7c0aba30 con 0x7fee7c1430c0 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee74001b90 con 0x7fee7c1430c0 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 10 monclient(hunting): dump: 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:06.315 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:06.316 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.314+0000 7fee727fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:06.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3779409793 0 0) 0x7fee7c0aba30 con 0x7fee7c1430c0 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 >> v1:172.21.15.161:6789/0 conn(0x7fee7c0a98f0 legacy=0x7fee7c138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 >> v1:172.21.15.132:6789/0 conn(0x7fee7c13f710 legacy=0x7fee7c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 1 monclient: found mon.c 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7c1d8340 con 0x7fee7c1430c0 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 10 monclient: _renew_subs 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:06.323 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee7c134820 con 0x7fee7c1430c0 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 10 monclient: _renew_subs 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee7c1380c0 con 0x7fee7c1430c0 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.321+0000 7fee83ffb640 1 librados: init done 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.322+0000 7fee83ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.322+0000 7fee83ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.324 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.322+0000 7fee83ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:06.325 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.322+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee7c0627b0 con 0x7fee7c1430c0 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee740020c0 con 0x7fee7c1430c0 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fee74003b30 con 0x7fee7c1430c0 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee727fc640 10 monclient: dump: 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:06.620 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.618+0000 7fee73fff640 4 set_mon_vals no callback set 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.619+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fee74003d10 con 0x7fee7c1430c0 2024-03-25T18:24:06.621 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.620+0000 7fee81d97640 10 monclient: get_auth_request con 0x7fee6006dd10 auth_method 0 2024-03-25T18:24:06.622 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.620+0000 7fee727fc640 10 client.4697.objecter ms_handle_connect 0x7fee6006dd10 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7fee740538c0 con 0x7fee7c1430c0 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 10 client.4697.objecter ms_dispatch 0x7fee7c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee7404f200 con 0x7fee7c1430c0 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.627 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.625+0000 7fee727fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:06.880 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.879+0000 7fee83ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.880 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.879+0000 7fee83ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.880 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.879+0000 7fee83ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:06.880 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.879+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee7c14e8b0 con 0x7fee7c1430c0 2024-03-25T18:24:06.923 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.922+0000 7fee727fc640 1 -- 172.21.15.132:0/2474154609 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7fee74051300 con 0x7fee7c1430c0 2024-03-25T18:24:06.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.922+0000 7fee727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.924 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.922+0000 7fee727fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:06.924 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:06.924 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 10 librados: watch_flush enter 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 10 librados: watch_flush exit 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 10 client.4697.objecter successfully canceled tick 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 >> v1:172.21.15.161:6800/411950978 conn(0x7fee6006dd10 legacy=0x7fee600701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 10 monclient: shutdown 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 >> v1:172.21.15.132:6790/0 conn(0x7fee7c1430c0 legacy=0x7fee7c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 shutdown_connections 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 >> 172.21.15.132:0/2474154609 conn(0x7fee7c0b7ea0 msgr2=0x7fee7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 shutdown_connections 2024-03-25T18:24:06.927 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 -- 172.21.15.132:0/2474154609 wait complete. 2024-03-25T18:24:06.928 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:06.925+0000 7fee83ffb640 1 librados: shutdown 2024-03-25T18:24:06.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:13.744 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.860+0000 7f18e3390640 10 monclient: get_monmap_and_config 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.860+0000 7f18e3390640 10 monclient: build_initial_monmap 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.860+0000 7f18e3390640 10 monclient: monmap: 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:13.860127+0000 2024-03-25T18:24:13.860 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:13.860127+0000 2024-03-25T18:24:13.861 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:13.861 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:13.861 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:13.861 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:13.861 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:13.861 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:13.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.860+0000 7f18e3390640 1 Processor -- start 2024-03-25T18:24:13.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 1 -- start start 2024-03-25T18:24:13.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 10 monclient: init 2024-03-25T18:24:13.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:13.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:13.862 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 10 monclient(hunting): picked mon.noname-b con 0x7f18dc0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 10 monclient(hunting): picked mon.noname-a con 0x7f18dc0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.861+0000 7f18e3390640 10 monclient(hunting): picked mon.noname-c con 0x7f18dc14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.862+0000 7f18e3390640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc0af940 con 0x7f18dc0b1730 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.862+0000 7f18e3390640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc0b0a70 con 0x7f18dc14be90 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.862+0000 7f18e3390640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc0b0d50 con 0x7f18dc0b4d60 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.862+0000 7f18e3390640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.863 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.862+0000 7f18e3390640 10 monclient(hunting): authenticate will time out at 7894.124287s 2024-03-25T18:24:13.877 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.877+0000 7f18e192d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f18dc14be90 0x7f18dc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41320/0 (socket says 172.21.15.132:41320) 2024-03-25T18:24:13.877 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.877+0000 7f18e192d640 1 -- 172.21.15.132:0/2682493584 learned_addr learned my addr 172.21.15.132:0/2682493584 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:13.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.890+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525484108 0 0) 0x7f18dc0b0a70 con 0x7f18dc14be90 2024-03-25T18:24:13.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.891+0000 7f18d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.891+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.891+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.891+0000 7f18d3fff640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:24:13.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.891+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c8003520 con 0x7f18dc14be90 2024-03-25T18:24:13.898 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.898+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660412269 0 0) 0x7f18dc0af940 con 0x7f18dc0b1730 2024-03-25T18:24:13.898 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.898+0000 7f18d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.898+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.898+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.898+0000 7f18d3fff640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:24:13.899 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.898+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc0b0a70 con 0x7f18dc0b1730 2024-03-25T18:24:13.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.902+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040316896 0 0) 0x7f18dc0b0d50 con 0x7f18dc0b4d60 2024-03-25T18:24:13.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.902+0000 7f18d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.902+0000 7f18d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.902 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.902+0000 7f18d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.902+0000 7f18d3fff640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:24:13.903 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.902+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc0af940 con 0x7f18dc0b4d60 2024-03-25T18:24:13.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3393541247 0 0) 0x7f18dc0af940 con 0x7f18dc0b4d60 2024-03-25T18:24:13.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 >> v1:172.21.15.161:6789/0 conn(0x7f18dc14be90 legacy=0x7f18dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 >> v1:172.21.15.132:6789/0 conn(0x7f18dc0b1730 legacy=0x7f18dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:13.914 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 1 monclient: found mon.noname-b 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc0b1030 con 0x7f18dc0b4d60 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f18d80015b0 con 0x7f18dc0b4d60 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18d3fff640 10 monclient: dump: 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.915 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18e3390640 5 monclient: authenticate success, global_id 4700 2024-03-25T18:24:13.916 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.914+0000 7f18e3390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:13.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d8001710 con 0x7f18dc0b4d60 2024-03-25T18:24:13.925 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18d3fff640 1 -- 172.21.15.132:0/2682493584 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f18d8001b60 con 0x7f18dc0b4d60 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18e3390640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18e3390640 4 set_mon_vals no callback set 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.925+0000 7f18d3fff640 10 monclient: dump: 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:13.926 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:13.927 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:13.927 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 10 monclient: shutdown 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 1 -- 172.21.15.132:0/2682493584 >> v1:172.21.15.132:6790/0 conn(0x7f18dc0b4d60 legacy=0x7f18dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 1 -- 172.21.15.132:0/2682493584 shutdown_connections 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 1 -- 172.21.15.132:0/2682493584 >> 172.21.15.132:0/2682493584 conn(0x7f18dc0617a0 msgr2=0x7f18dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 1 -- 172.21.15.132:0/2682493584 shutdown_connections 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.926+0000 7f18e3390640 1 -- 172.21.15.132:0/2682493584 wait complete. 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 10 monclient: build_initial_monmap 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 10 monclient: monmap: 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:13.929 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 1 librados: starting msgr at 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 1 librados: starting objecter 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 1 Processor -- start 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 1 -- start start 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 1 librados: setting wanted keys 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 1 librados: calling monclient init 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.927+0000 7f18e3390640 10 monclient: init 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient(hunting): picked mon.noname-b con 0x7f18dc0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:13.930 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient(hunting): picked mon.noname-c con 0x7f18dc0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:13.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient(hunting): picked mon.noname-a con 0x7f18dc14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:13.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc13e200 con 0x7f18dc14be90 2024-03-25T18:24:13.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc12f5f0 con 0x7f18dc0b1730 2024-03-25T18:24:13.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18dc12f780 con 0x7f18dc0b4d60 2024-03-25T18:24:13.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.931 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.928+0000 7f18e3390640 10 monclient(hunting): authenticate will time out at 7894.189845s 2024-03-25T18:24:13.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.939+0000 7f18e112c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f18dc0b1730 0x7f18dc145330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:41386/0 (socket says 172.21.15.132:41386) 2024-03-25T18:24:13.939 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.939+0000 7f18e112c640 1 -- 172.21.15.132:0/1709131403 learned_addr learned my addr 172.21.15.132:0/1709131403 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.941+0000 7f18d1ffb640 10 client.?.objecter ms_handle_connect 0x7f18dc0b1730 2024-03-25T18:24:13.941 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.941+0000 7f18d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.942+0000 7f18d1ffb640 10 client.?.objecter ms_handle_connect 0x7f18dc14be90 2024-03-25T18:24:13.942 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.942+0000 7f18d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.948+0000 7f18d1ffb640 10 client.?.objecter ms_handle_connect 0x7f18dc0b4d60 2024-03-25T18:24:13.948 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.948+0000 7f18d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.952+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948007282 0 0) 0x7f18dc12f5f0 con 0x7f18dc0b1730 2024-03-25T18:24:13.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.952+0000 7f18d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.952 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.952+0000 7f18d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.952+0000 7f18d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.952+0000 7f18d1ffb640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.952+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c0004eb0 con 0x7f18dc0b1730 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.953+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717580788 0 0) 0x7f18dc13e200 con 0x7f18dc14be90 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.953+0000 7f18d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.953+0000 7f18d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.953+0000 7f18d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.953 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.953+0000 7f18d1ffb640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:24:13.954 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.953+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc12f5f0 con 0x7f18dc14be90 2024-03-25T18:24:13.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3237120725 0 0) 0x7f18dc12f5f0 con 0x7f18dc14be90 2024-03-25T18:24:13.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18dc13e200 con 0x7f18dc14be90 2024-03-25T18:24:13.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f18d4001ab0 con 0x7f18dc14be90 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:13.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.969+0000 7f18d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:13.973 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.973+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001039017 0 0) 0x7f18dc12f780 con 0x7f18dc0b4d60 2024-03-25T18:24:13.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.973+0000 7f18d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.973+0000 7f18d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.973+0000 7f18d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.973+0000 7f18d1ffb640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:24:13.974 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.973+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc12f5f0 con 0x7f18dc0b4d60 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.983+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 669942398 0 0) 0x7f18c0004eb0 con 0x7f18dc0b1730 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.983+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18dc12f780 con 0x7f18dc0b1730 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.983+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f18d8001b20 con 0x7f18dc0b1730 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.983+0000 7f18d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.983+0000 7f18d1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.983+0000 7f18d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.984 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:13.985 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:13.985 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:13.985 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:13.985 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1307819963 0 0) 0x7f18dc13e200 con 0x7f18dc14be90 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 >> v1:172.21.15.132:6790/0 conn(0x7f18dc0b4d60 legacy=0x7f18dc12eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 >> v1:172.21.15.161:6789/0 conn(0x7f18dc0b1730 legacy=0x7f18dc145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 1 monclient: found mon.a 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:13.994 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc13dd20 con 0x7f18dc14be90 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 5 monclient: authenticate success, global_id 4686 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 10 monclient: _renew_subs 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc13df70 con 0x7f18dc14be90 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 10 client.4686.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 10 monclient: _renew_subs 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc135f50 con 0x7f18dc14be90 2024-03-25T18:24:13.995 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.993+0000 7f18e3390640 1 librados: init done 2024-03-25T18:24:13.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.994+0000 7f18e3390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.994+0000 7f18e3390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.994+0000 7f18e3390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:24:13.996 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:13.994+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc061ca0 con 0x7f18dc14be90 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18d4001fa0 con 0x7f18dc14be90 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f18d4004d00 con 0x7f18dc14be90 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d1ffb640 10 monclient: dump: 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:14.320 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.319+0000 7f18d37fe640 4 set_mon_vals no callback set 2024-03-25T18:24:14.321 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.320+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f18d401acd0 con 0x7f18dc14be90 2024-03-25T18:24:14.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.321+0000 7f18e112c640 10 monclient: get_auth_request con 0x7f18c006df20 auth_method 0 2024-03-25T18:24:14.322 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.321+0000 7f18d1ffb640 10 client.4686.objecter ms_handle_connect 0x7f18c006df20 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.327+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f18d40548c0 con 0x7f18dc14be90 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.327+0000 7f18d1ffb640 10 client.4686.objecter ms_dispatch 0x7f18dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.327+0000 7f18d1ffb640 3 client.4686.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.327+0000 7f18d1ffb640 3 client.4686.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.327+0000 7f18d1ffb640 20 client.4686.objecter dump_active .. 0 homeless 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.328+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18d4050200 con 0x7f18dc14be90 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.328+0000 7f18d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:14.328 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.328+0000 7f18d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:14.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.567+0000 7f18e3390640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:14.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.567+0000 7f18e3390640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:14.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.567+0000 7f18e3390640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.132:6789/0 2024-03-25T18:24:14.569 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.567+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 --> v1:172.21.15.132:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18dc0627b0 con 0x7f18dc14be90 2024-03-25T18:24:14.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.614+0000 7f18d1ffb640 1 -- 172.21.15.132:0/1709131403 <== mon.0 v1:172.21.15.132:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f18d4052300 con 0x7f18dc14be90 2024-03-25T18:24:14.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.614+0000 7f18d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:14.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.614+0000 7f18d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:14.615 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:14.616 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.616+0000 7f18e3390640 10 librados: watch_flush enter 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.616+0000 7f18e3390640 10 librados: watch_flush exit 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.616+0000 7f18e3390640 20 client.4686.objecter shutdown clearing up homeless session... 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.616+0000 7f18e3390640 10 client.4686.objecter successfully canceled tick 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 >> v1:172.21.15.161:6800/411950978 conn(0x7f18c006df20 legacy=0x7f18c00703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 10 monclient: shutdown 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 >> v1:172.21.15.132:6789/0 conn(0x7f18dc14be90 legacy=0x7f18dc1379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 shutdown_connections 2024-03-25T18:24:14.619 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 >> 172.21.15.132:0/1709131403 conn(0x7f18dc0617a0 msgr2=0x7f18dc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:14.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 shutdown_connections 2024-03-25T18:24:14.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 -- 172.21.15.132:0/1709131403 wait complete. 2024-03-25T18:24:14.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:14.617+0000 7f18e3390640 1 librados: shutdown 2024-03-25T18:24:14.631 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:15.943 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:01.207 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:15.990 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:01.254 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:15.992 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:01.256 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:15.995 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:01.259 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:16.032 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:01.297 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:16.047 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:01.311 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:16.080 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:01.322 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:16.087 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:01.329 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:16.104 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:01.345 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:21.438 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:21.556 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.555+0000 7f0d43a35640 10 monclient: get_monmap_and_config 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.555+0000 7f0d43a35640 10 monclient: build_initial_monmap 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.555+0000 7f0d43a35640 10 monclient: monmap: 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:21.556704+0000 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:21.556704+0000 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:21.558 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.558 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:21.558 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:21.558 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:21.558 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.556+0000 7f0d43a35640 1 Processor -- start 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.556+0000 7f0d43a35640 1 -- start start 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.556+0000 7f0d43a35640 10 monclient: init 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c0b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c0b1730 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c0af940 con 0x7f0d3c14be90 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c0b0a70 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c0b0d50 con 0x7f0d3c0b1730 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:21.561 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.557+0000 7f0d43a35640 10 monclient(hunting): authenticate will time out at 7901.820253s 2024-03-25T18:24:21.567 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.566+0000 7f0d40fd0640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f0d3c0b1730 0x7f0d3c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:46004/0 (socket says 172.21.15.132:46004) 2024-03-25T18:24:21.568 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.566+0000 7f0d40fd0640 1 -- 172.21.15.132:0/3193923064 learned_addr learned my addr 172.21.15.132:0/3193923064 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:21.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.577+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963679113 0 0) 0x7f0d3c0b0d50 con 0x7f0d3c0b1730 2024-03-25T18:24:21.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.577+0000 7f0d33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.577+0000 7f0d33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.577+0000 7f0d33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.578 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.577+0000 7f0d33fff640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:24:21.579 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.577+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d28003520 con 0x7f0d3c0b1730 2024-03-25T18:24:21.591 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002516759 0 0) 0x7f0d28003520 con 0x7f0d3c0b1730 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 >> v1:172.21.15.161:6789/0 conn(0x7f0d3c0b4d60 legacy=0x7f0d3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 >> v1:172.21.15.132:6789/0 conn(0x7f0d3c14be90 legacy=0x7f0d3c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 1 monclient: found mon.noname-b 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.132:6790/0 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c0b1030 con 0x7f0d3c0b1730 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:21.592 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0d2c001690 con 0x7f0d3c0b1730 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d33fff640 10 monclient: dump: 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:21.593 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:21.594 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d43a35640 5 monclient: authenticate success, global_id 4706 2024-03-25T18:24:21.594 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.590+0000 7f0d43a35640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 <== mon.2 v1:172.21.15.132:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d2c0017f0 con 0x7f0d3c0b1730 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d33fff640 1 -- 172.21.15.132:0/3193923064 <== mon.2 v1:172.21.15.132:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0d2c001c40 con 0x7f0d3c0b1730 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d43a35640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d43a35640 4 set_mon_vals no callback set 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.606+0000 7f0d33fff640 10 monclient: dump: 2024-03-25T18:24:21.608 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:21.609 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 10 monclient: shutdown 2024-03-25T18:24:21.612 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 1 -- 172.21.15.132:0/3193923064 >> v1:172.21.15.132:6790/0 conn(0x7f0d3c0b1730 legacy=0x7f0d3c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 1 -- 172.21.15.132:0/3193923064 shutdown_connections 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 1 -- 172.21.15.132:0/3193923064 >> 172.21.15.132:0/3193923064 conn(0x7f0d3c0617a0 msgr2=0x7f0d3c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 1 -- 172.21.15.132:0/3193923064 shutdown_connections 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.607+0000 7f0d43a35640 1 -- 172.21.15.132:0/3193923064 wait complete. 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.608+0000 7f0d43a35640 10 monclient: build_initial_monmap 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.608+0000 7f0d43a35640 10 monclient: monmap: 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:21.613 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 1 librados: starting msgr at 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 1 librados: starting objecter 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 1 Processor -- start 2024-03-25T18:24:21.614 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 1 -- start start 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 1 librados: setting wanted keys 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 1 librados: calling monclient init 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 10 monclient: init 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.609+0000 7f0d43a35640 10 monclient(hunting): picked mon.noname-c con 0x7f0d3c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.610+0000 7f0d43a35640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.610+0000 7f0d43a35640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c1dafb0 con 0x7f0d3c14be90 2024-03-25T18:24:21.615 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.610+0000 7f0d43a35640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c1432d0 con 0x7f0d3c0b1730 2024-03-25T18:24:21.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.610+0000 7f0d43a35640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c1dbbe0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.610+0000 7f0d43a35640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:21.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.610+0000 7f0d43a35640 10 monclient(hunting): authenticate will time out at 7901.872901s 2024-03-25T18:24:21.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.614+0000 7f0d41fd2640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f0d3c14be90 0x7f0d3c13b4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:47500/0 (socket says 172.21.15.132:47500) 2024-03-25T18:24:21.616 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.614+0000 7f0d41fd2640 1 -- 172.21.15.132:0/3758488466 learned_addr learned my addr 172.21.15.132:0/3758488466 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:21.617 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.614+0000 7f0d31ffb640 10 client.?.objecter ms_handle_connect 0x7f0d3c14be90 2024-03-25T18:24:21.617 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.614+0000 7f0d31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.619+0000 7f0d31ffb640 10 client.?.objecter ms_handle_connect 0x7f0d3c0b4d60 2024-03-25T18:24:21.620 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.619+0000 7f0d31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:21.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.622+0000 7f0d31ffb640 10 client.?.objecter ms_handle_connect 0x7f0d3c0b1730 2024-03-25T18:24:21.623 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.622+0000 7f0d31ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.623+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058476230 0 0) 0x7f0d3c1dbbe0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.623+0000 7f0d31ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.623+0000 7f0d31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.623+0000 7f0d31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.623+0000 7f0d31ffb640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:24:21.625 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.623+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d20004eb0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.626+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724189072 0 0) 0x7f0d3c1dafb0 con 0x7f0d3c14be90 2024-03-25T18:24:21.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.626+0000 7f0d31ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.626+0000 7f0d31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.626+0000 7f0d31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.626+0000 7f0d31ffb640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:24:21.628 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.627+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d3c1dbbe0 con 0x7f0d3c14be90 2024-03-25T18:24:21.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.635+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876439448 0 0) 0x7f0d3c1432d0 con 0x7f0d3c0b1730 2024-03-25T18:24:21.636 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.635+0000 7f0d31ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:21.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.635+0000 7f0d31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:21.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.635+0000 7f0d31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:21.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.635+0000 7f0d31ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:24:21.637 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.635+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d3c1dafb0 con 0x7f0d3c0b1730 2024-03-25T18:24:21.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3310051998 0 0) 0x7f0d20004eb0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d3c1432d0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.639 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0d2c001b00 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.640 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:21.641 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:21.641 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:21.641 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.641 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.638+0000 7f0d31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.642+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077174780 0 0) 0x7f0d3c1dbbe0 con 0x7f0d3c14be90 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.642+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d20004eb0 con 0x7f0d3c14be90 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.642+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0d34001c10 con 0x7f0d3c14be90 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.642+0000 7f0d31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.642+0000 7f0d31ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.642+0000 7f0d31ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.644 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:21.645 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.656+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539328970 0 0) 0x7f0d3c1dafb0 con 0x7f0d3c0b1730 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.656+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d3c1dbbe0 con 0x7f0d3c0b1730 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.656+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0d38001860 con 0x7f0d3c0b1730 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.656+0000 7f0d31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.656+0000 7f0d31ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.656+0000 7f0d31ffb640 10 monclient(hunting): dump: 2024-03-25T18:24:21.657 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:21.658 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.658+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2388228587 0 0) 0x7f0d3c1432d0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.658+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 >> v1:172.21.15.161:6789/0 conn(0x7f0d3c0b1730 legacy=0x7f0d3c1db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.658+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 >> v1:172.21.15.132:6789/0 conn(0x7f0d3c14be90 legacy=0x7f0d3c13b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.658+0000 7f0d31ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.658+0000 7f0d31ffb640 1 monclient: found mon.c 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c139670 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d31ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:21.661 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d31ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 5 monclient: authenticate success, global_id 4709 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 10 monclient: _renew_subs 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c13bbe0 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 10 monclient: _renew_subs 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c142a30 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.662 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.659+0000 7f0d43a35640 1 librados: init done 2024-03-25T18:24:21.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.660+0000 7f0d43a35640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.660+0000 7f0d43a35640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.660+0000 7f0d43a35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:21.663 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.660+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c1db140 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d2c001d20 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:21.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f0d2c004d90 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.969 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d31ffb640 10 monclient: dump: 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:21.970 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:21.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.968+0000 7f0d337fe640 4 set_mon_vals no callback set 2024-03-25T18:24:21.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.969+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f0d2c01ad60 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.970+0000 7f0d417d1640 10 monclient: get_auth_request con 0x7f0d2006e010 auth_method 0 2024-03-25T18:24:21.971 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.970+0000 7f0d31ffb640 10 client.4709.objecter ms_handle_connect 0x7f0d2006e010 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f0d2c054950 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 10 client.4709.objecter ms_dispatch 0x7f0d3c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 3 client.4709.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 3 client.4709.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d2c050290 con 0x7f0d3c0b4d60 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.976 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:21.974+0000 7f0d31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:22.221 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.220+0000 7f0d43a35640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:22.222 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.220+0000 7f0d43a35640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:22.222 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.220+0000 7f0d43a35640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:22.222 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.220+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d3c0627b0 con 0x7f0d3c0b4d60 2024-03-25T18:24:22.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.246+0000 7f0d31ffb640 1 -- 172.21.15.132:0/3758488466 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f0d2c052390 con 0x7f0d3c0b4d60 2024-03-25T18:24:22.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.246+0000 7f0d31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:22.247 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.246+0000 7f0d31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:22.247 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:22.248 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 10 librados: watch_flush enter 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 10 librados: watch_flush exit 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 10 client.4709.objecter successfully canceled tick 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 >> v1:172.21.15.161:6800/411950978 conn(0x7f0d2006e010 legacy=0x7f0d200704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 10 monclient: shutdown 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:22.250 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 >> v1:172.21.15.132:6790/0 conn(0x7f0d3c0b4d60 legacy=0x7f0d3c138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 shutdown_connections 2024-03-25T18:24:22.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 >> 172.21.15.132:0/3758488466 conn(0x7f0d3c0617a0 msgr2=0x7f0d3c137300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:22.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 shutdown_connections 2024-03-25T18:24:22.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.248+0000 7f0d43a35640 1 -- 172.21.15.132:0/3758488466 wait complete. 2024-03-25T18:24:22.251 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:22.249+0000 7f0d43a35640 1 librados: shutdown 2024-03-25T18:24:22.259 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:28.276 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:13.540 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.283 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:13.547 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.293 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:13.557 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.293 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:13.535 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.296 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:13.537 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.297 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:13.562 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.308 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:13.572 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.313 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:13.554 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:28.315 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:13.579 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:29.066 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:29.179 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.178+0000 7f3b5e2ae640 10 monclient: get_monmap_and_config 2024-03-25T18:24:29.179 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.178+0000 7f3b5e2ae640 10 monclient: build_initial_monmap 2024-03-25T18:24:29.179 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.178+0000 7f3b5e2ae640 10 monclient: monmap: 2024-03-25T18:24:29.179 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:29.179284+0000 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:29.179284+0000 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:29.180 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:29.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.179+0000 7f3b5e2ae640 1 Processor -- start 2024-03-25T18:24:29.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.179+0000 7f3b5e2ae640 1 -- start start 2024-03-25T18:24:29.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.179+0000 7f3b5e2ae640 10 monclient: init 2024-03-25T18:24:29.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:29.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:29.181 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3b580a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3b581430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b58142520 con 0x7f3b5813f710 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580b28a0 con 0x7f3b580a98f0 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580b2a30 con 0x7f3b581430c0 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.182 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.180+0000 7f3b5e2ae640 10 monclient(hunting): authenticate will time out at 7909.443054s 2024-03-25T18:24:29.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.189+0000 7f3b5c84b640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f3b581430c0 0x7f3b5814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:38834/0 (socket says 172.21.15.132:38834) 2024-03-25T18:24:29.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.189+0000 7f3b5c84b640 1 -- 172.21.15.132:0/3039790919 learned_addr learned my addr 172.21.15.132:0/3039790919 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:29.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.209+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019882513 0 0) 0x7f3b580b2a30 con 0x7f3b581430c0 2024-03-25T18:24:29.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.209+0000 7f3b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.209+0000 7f3b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.209+0000 7f3b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.209+0000 7f3b56ffd640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:24:29.211 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.209+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b40003520 con 0x7f3b581430c0 2024-03-25T18:24:29.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.214+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717711478 0 0) 0x7f3b580b28a0 con 0x7f3b580a98f0 2024-03-25T18:24:29.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.214+0000 7f3b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.214+0000 7f3b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.214+0000 7f3b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.214+0000 7f3b56ffd640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:24:29.215 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.214+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b580b2a30 con 0x7f3b580a98f0 2024-03-25T18:24:29.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.222+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431809287 0 0) 0x7f3b58142520 con 0x7f3b5813f710 2024-03-25T18:24:29.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.222+0000 7f3b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.222+0000 7f3b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.222+0000 7f3b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.222+0000 7f3b56ffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:24:29.224 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.222+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b580b28a0 con 0x7f3b5813f710 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3736186625 0 0) 0x7f3b580b2a30 con 0x7f3b580a98f0 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 >> v1:172.21.15.132:6790/0 conn(0x7f3b581430c0 legacy=0x7f3b5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 >> v1:172.21.15.132:6789/0 conn(0x7f3b5813f710 legacy=0x7f3b581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 1 monclient: found mon.noname-c 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580b44c0 con 0x7f3b580a98f0 2024-03-25T18:24:29.227 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3b48001670 con 0x7f3b580a98f0 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b56ffd640 10 monclient: dump: 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:29.228 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:29.229 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:29.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b5e2ae640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:24:29.229 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.225+0000 7f3b5e2ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.266+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b480017d0 con 0x7f3b580a98f0 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.266+0000 7f3b56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.266+0000 7f3b56ffd640 1 -- 172.21.15.132:0/3039790919 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3b48001be0 con 0x7f3b580a98f0 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.266+0000 7f3b5e2ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.266+0000 7f3b5e2ae640 4 set_mon_vals no callback set 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.266+0000 7f3b56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b56ffd640 10 monclient: dump: 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:29.268 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:29.269 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:29.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 10 monclient: shutdown 2024-03-25T18:24:29.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/3039790919 >> v1:172.21.15.161:6789/0 conn(0x7f3b580a98f0 legacy=0x7f3b580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/3039790919 shutdown_connections 2024-03-25T18:24:29.271 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/3039790919 >> 172.21.15.132:0/3039790919 conn(0x7f3b580b7ea0 msgr2=0x7f3b580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/3039790919 shutdown_connections 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.267+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/3039790919 wait complete. 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.268+0000 7f3b5e2ae640 10 monclient: build_initial_monmap 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.268+0000 7f3b5e2ae640 10 monclient: monmap: 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.268+0000 7f3b5e2ae640 1 librados: starting msgr at 2024-03-25T18:24:29.272 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.268+0000 7f3b5e2ae640 1 librados: starting objecter 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.268+0000 7f3b5e2ae640 1 Processor -- start 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 1 -- start start 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 1 librados: setting wanted keys 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 1 librados: calling monclient init 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient: init 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient(hunting): picked mon.noname-b con 0x7f3b580a98f0 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient(hunting): picked mon.noname-a con 0x7f3b5813f710 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient(hunting): picked mon.noname-c con 0x7f3b581430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b58139330 con 0x7f3b5813f710 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580abe70 con 0x7f3b580a98f0 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b580ac150 con 0x7f3b581430c0 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:29.273 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.269+0000 7f3b5e2ae640 10 monclient(hunting): authenticate will time out at 7909.532435s 2024-03-25T18:24:29.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.280+0000 7f3b57fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3b580a98f0 0x7f3b581389d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:56168/0 (socket says 172.21.15.132:56168) 2024-03-25T18:24:29.281 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.280+0000 7f3b57fff640 1 -- 172.21.15.132:0/2626156087 learned_addr learned my addr 172.21.15.132:0/2626156087 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:29.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.283+0000 7f3b54ff9640 10 client.?.objecter ms_handle_connect 0x7f3b580a98f0 2024-03-25T18:24:29.284 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.283+0000 7f3b54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.289+0000 7f3b54ff9640 10 client.?.objecter ms_handle_connect 0x7f3b5813f710 2024-03-25T18:24:29.290 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.289+0000 7f3b54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.292+0000 7f3b54ff9640 10 client.?.objecter ms_handle_connect 0x7f3b581430c0 2024-03-25T18:24:29.293 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.292+0000 7f3b54ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:29.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.294+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772602380 0 0) 0x7f3b580abe70 con 0x7f3b580a98f0 2024-03-25T18:24:29.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.294+0000 7f3b54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.294+0000 7f3b54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.294+0000 7f3b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.294+0000 7f3b54ff9640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:24:29.296 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.295+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b44004eb0 con 0x7f3b580a98f0 2024-03-25T18:24:29.307 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 214928920 0 0) 0x7f3b44004eb0 con 0x7f3b580a98f0 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b580abe70 con 0x7f3b580a98f0 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3b48001670 con 0x7f3b580a98f0 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:29.308 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:29.309 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:29.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.306+0000 7f3b54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:29.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.309+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055696423 0 0) 0x7f3b58139330 con 0x7f3b5813f710 2024-03-25T18:24:29.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.309+0000 7f3b54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.310 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.309+0000 7f3b54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.309+0000 7f3b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.309+0000 7f3b54ff9640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:24:29.311 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.309+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b44004eb0 con 0x7f3b5813f710 2024-03-25T18:24:29.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.319+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846205686 0 0) 0x7f3b580ac150 con 0x7f3b581430c0 2024-03-25T18:24:29.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.319+0000 7f3b54ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:29.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.319+0000 7f3b54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:29.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.319+0000 7f3b54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:29.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.319+0000 7f3b54ff9640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:24:29.320 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.319+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b58139330 con 0x7f3b581430c0 2024-03-25T18:24:29.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3074704402 0 0) 0x7f3b580abe70 con 0x7f3b580a98f0 2024-03-25T18:24:29.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 >> v1:172.21.15.132:6790/0 conn(0x7f3b581430c0 legacy=0x7f3b581e2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 >> v1:172.21.15.132:6789/0 conn(0x7f3b5813f710 legacy=0x7f3b580ac570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:29.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 1 monclient: found mon.b 2024-03-25T18:24:29.335 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b580acc80 con 0x7f3b580a98f0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 10 monclient: _renew_subs 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b58137e30 con 0x7f3b580a98f0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 10 monclient: _renew_subs 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b58136880 con 0x7f3b580a98f0 2024-03-25T18:24:29.336 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.333+0000 7f3b5e2ae640 1 librados: init done 2024-03-25T18:24:29.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.335+0000 7f3b5e2ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.335+0000 7f3b5e2ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.335+0000 7f3b5e2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:29.337 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.335+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b580627b0 con 0x7f3b580a98f0 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b480038a0 con 0x7f3b580a98f0 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f3b48003cf0 con 0x7f3b580a98f0 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b54ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b54ff9640 10 monclient: dump: 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:29.604 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:29.605 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.602+0000 7f3b55ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:29.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.603+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f3b48003f10 con 0x7f3b580a98f0 2024-03-25T18:24:29.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.604+0000 7f3b577fe640 10 monclient: get_auth_request con 0x7f3b4406de80 auth_method 0 2024-03-25T18:24:29.606 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.604+0000 7f3b54ff9640 10 client.4723.objecter ms_handle_connect 0x7f3b4406de80 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.607+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f3b48054990 con 0x7f3b580a98f0 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.607+0000 7f3b54ff9640 10 client.4723.objecter ms_dispatch 0x7f3b58000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.607+0000 7f3b54ff9640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.607+0000 7f3b54ff9640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.607+0000 7f3b54ff9640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.608+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b480504a0 con 0x7f3b580a98f0 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.608+0000 7f3b54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:29.609 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.608+0000 7f3b54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:29.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.819+0000 7f3b5e2ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.819+0000 7f3b5e2ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.819+0000 7f3b5e2ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:29.820 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.819+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b581390e0 con 0x7f3b580a98f0 2024-03-25T18:24:29.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.848+0000 7f3b54ff9640 1 -- 172.21.15.132:0/2626156087 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f3b480523d0 con 0x7f3b580a98f0 2024-03-25T18:24:29.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.848+0000 7f3b54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.848+0000 7f3b54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:29.850 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:29.850 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:29.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.850+0000 7f3b5e2ae640 10 librados: watch_flush enter 2024-03-25T18:24:29.852 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.850+0000 7f3b5e2ae640 10 librados: watch_flush exit 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.850+0000 7f3b5e2ae640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.850+0000 7f3b5e2ae640 10 client.4723.objecter successfully canceled tick 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 >> v1:172.21.15.161:6800/411950978 conn(0x7f3b4406de80 legacy=0x7f3b44070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 10 monclient: shutdown 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 >> v1:172.21.15.161:6789/0 conn(0x7f3b580a98f0 legacy=0x7f3b581389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 shutdown_connections 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 >> 172.21.15.132:0/2626156087 conn(0x7f3b580b7ea0 msgr2=0x7f3b58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 shutdown_connections 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 -- 172.21.15.132:0/2626156087 wait complete. 2024-03-25T18:24:29.853 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:29.851+0000 7f3b5e2ae640 1 librados: shutdown 2024-03-25T18:24:29.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:36.003 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:21.267 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.049 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:21.313 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.052 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:21.316 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.057 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:21.321 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.089 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:21.353 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.093 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:21.357 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.114 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:21.356 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.129 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:21.371 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.151 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:21.392 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.671 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:36.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.785+0000 7f2a0f138640 10 monclient: get_monmap_and_config 2024-03-25T18:24:36.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.785+0000 7f2a0f138640 10 monclient: build_initial_monmap 2024-03-25T18:24:36.786 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.785+0000 7f2a0f138640 10 monclient: monmap: 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:36.786435+0000 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:36.786435+0000 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:36.787 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.788 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 1 Processor -- start 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 1 -- start start 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 10 monclient: init 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 10 monclient(hunting): picked mon.noname-a con 0x7f2a080a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.786+0000 7f2a0f138640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.787+0000 7f2a0f138640 10 monclient(hunting): picked mon.noname-b con 0x7f2a081430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.787+0000 7f2a0f138640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a08142520 con 0x7f2a080a98f0 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.787+0000 7f2a0f138640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080b28a0 con 0x7f2a0813f710 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.787+0000 7f2a0f138640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a080b2a30 con 0x7f2a081430c0 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.787+0000 7f2a0f138640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:36.789 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.787+0000 7f2a0f138640 10 monclient(hunting): authenticate will time out at 7917.049860s 2024-03-25T18:24:36.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.790+0000 7f2a0ced4640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f2a080a98f0 0x7f2a080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39892/0 (socket says 172.21.15.132:39892) 2024-03-25T18:24:36.791 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.790+0000 7f2a0ced4640 1 -- 172.21.15.132:0/1371787486 learned_addr learned my addr 172.21.15.132:0/1371787486 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.806+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033535675 0 0) 0x7f2a080b28a0 con 0x7f2a0813f710 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.806+0000 7f2a077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.806+0000 7f2a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.806+0000 7f2a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.806+0000 7f2a077fe640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.806+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003520 con 0x7f2a0813f710 2024-03-25T18:24:36.809 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.808+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306159993 0 0) 0x7f2a08142520 con 0x7f2a080a98f0 2024-03-25T18:24:36.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.808+0000 7f2a077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.808+0000 7f2a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.808+0000 7f2a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.808+0000 7f2a077fe640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:24:36.810 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.808+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a080b28a0 con 0x7f2a080a98f0 2024-03-25T18:24:36.813 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.812+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167034185 0 0) 0x7f2a080b2a30 con 0x7f2a081430c0 2024-03-25T18:24:36.813 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.812+0000 7f2a077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.813 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.812+0000 7f2a077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.813 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.812+0000 7f2a077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.813 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.812+0000 7f2a077fe640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:24:36.813 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.812+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08142520 con 0x7f2a081430c0 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2406679774 0 0) 0x7f29f8003520 con 0x7f2a0813f710 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 >> v1:172.21.15.132:6790/0 conn(0x7f2a081430c0 legacy=0x7f2a0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 >> v1:172.21.15.132:6789/0 conn(0x7f2a080a98f0 legacy=0x7f2a080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 1 monclient: found mon.noname-c 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a080b44c0 con 0x7f2a0813f710 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:36.815 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29f4001690 con 0x7f2a0813f710 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a0f138640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a0f138640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a077fe640 10 monclient: dump: 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:36.816 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.817 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.813+0000 7f2a0f138640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:36.826 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29f40017f0 con 0x7f2a0813f710 2024-03-25T18:24:36.826 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a077fe640 1 -- 172.21.15.132:0/1371787486 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29f4001c00 con 0x7f2a0813f710 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a0f138640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a0f138640 4 set_mon_vals no callback set 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.825+0000 7f2a077fe640 10 monclient: dump: 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:36.827 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:36.828 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:36.828 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 10 monclient: shutdown 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 1 -- 172.21.15.132:0/1371787486 >> v1:172.21.15.161:6789/0 conn(0x7f2a0813f710 legacy=0x7f2a081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 1 -- 172.21.15.132:0/1371787486 shutdown_connections 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 1 -- 172.21.15.132:0/1371787486 >> 172.21.15.132:0/1371787486 conn(0x7f2a080b7ea0 msgr2=0x7f2a080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 1 -- 172.21.15.132:0/1371787486 shutdown_connections 2024-03-25T18:24:36.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.826+0000 7f2a0f138640 1 -- 172.21.15.132:0/1371787486 wait complete. 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.827+0000 7f2a0f138640 10 monclient: build_initial_monmap 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.827+0000 7f2a0f138640 10 monclient: monmap: 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.830 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 1 librados: starting msgr at 2024-03-25T18:24:36.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 1 librados: starting objecter 2024-03-25T18:24:36.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 1 Processor -- start 2024-03-25T18:24:36.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 1 -- start start 2024-03-25T18:24:36.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 1 librados: setting wanted keys 2024-03-25T18:24:36.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 1 librados: calling monclient init 2024-03-25T18:24:36.831 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.828+0000 7f2a0f138640 10 monclient: init 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient(hunting): picked mon.noname-a con 0x7f2a080a98f0 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0813f710 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient(hunting): picked mon.noname-c con 0x7f2a081430c0 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a08139070 con 0x7f2a080a98f0 2024-03-25T18:24:36.832 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a08133710 con 0x7f2a0813f710 2024-03-25T18:24:36.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a081338a0 con 0x7f2a081430c0 2024-03-25T18:24:36.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:36.833 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.829+0000 7f2a0f138640 10 monclient(hunting): authenticate will time out at 7917.092252s 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.842+0000 7f2a0ced4640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f2a080a98f0 0x7f2a081378b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:39912/0 (socket says 172.21.15.132:39912) 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.842+0000 7f2a0ced4640 1 -- 172.21.15.132:0/3883751457 learned_addr learned my addr 172.21.15.132:0/3883751457 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.842+0000 7f2a057fa640 10 client.?.objecter ms_handle_connect 0x7f2a080a98f0 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.842+0000 7f2a057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.848+0000 7f2a057fa640 10 client.?.objecter ms_handle_connect 0x7f2a0813f710 2024-03-25T18:24:36.849 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.848+0000 7f2a057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.854 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.853+0000 7f2a057fa640 10 client.?.objecter ms_handle_connect 0x7f2a081430c0 2024-03-25T18:24:36.855 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.853+0000 7f2a057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.864+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488806430 0 0) 0x7f2a081338a0 con 0x7f2a081430c0 2024-03-25T18:24:36.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.864+0000 7f2a057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.864+0000 7f2a057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.864+0000 7f2a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.864+0000 7f2a057fa640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:24:36.865 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.864+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29ec004eb0 con 0x7f2a081430c0 2024-03-25T18:24:36.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.866+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68244618 0 0) 0x7f2a08139070 con 0x7f2a080a98f0 2024-03-25T18:24:36.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.866+0000 7f2a057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.866+0000 7f2a057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.866+0000 7f2a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.866+0000 7f2a057fa640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:24:36.867 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.866+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a081338a0 con 0x7f2a080a98f0 2024-03-25T18:24:36.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.867+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 675677420 0 0) 0x7f2a08133710 con 0x7f2a0813f710 2024-03-25T18:24:36.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.867+0000 7f2a057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.867+0000 7f2a057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.867+0000 7f2a057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.867+0000 7f2a057fa640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:24:36.868 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.867+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a08139070 con 0x7f2a0813f710 2024-03-25T18:24:36.882 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1499262733 0 0) 0x7f29ec004eb0 con 0x7f2a081430c0 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a08133710 con 0x7f2a081430c0 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2a00001bd0 con 0x7f2a081430c0 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 10 monclient(hunting): dump: 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:36.883 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:36.884 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.884 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.881+0000 7f2a057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.883+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3109624791 0 0) 0x7f2a081338a0 con 0x7f2a080a98f0 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.883+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29ec004eb0 con 0x7f2a080a98f0 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.883+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29fc001970 con 0x7f2a080a98f0 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.883+0000 7f2a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.883+0000 7f2a057fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.883+0000 7f2a057fa640 10 monclient(hunting): dump: 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:36.885 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:36.886 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:36.886 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3221132910 0 0) 0x7f2a08139070 con 0x7f2a0813f710 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a081338a0 con 0x7f2a0813f710 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f29f4001b00 con 0x7f2a0813f710 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient(hunting): dump: 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.891 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3613942949 0 0) 0x7f2a08133710 con 0x7f2a081430c0 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 >> v1:172.21.15.161:6789/0 conn(0x7f2a0813f710 legacy=0x7f2a081daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 >> v1:172.21.15.132:6789/0 conn(0x7f2a080a98f0 legacy=0x7f2a081378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 monclient: found mon.c 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:36.892 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a08133a30 con 0x7f2a081430c0 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 5 monclient: authenticate success, global_id 4721 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 10 monclient: _renew_subs 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a080b7100 con 0x7f2a081430c0 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 10 client.4721.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 10 monclient: _renew_subs 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a08136e60 con 0x7f2a081430c0 2024-03-25T18:24:36.893 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.889+0000 7f2a0f138640 1 librados: init done 2024-03-25T18:24:36.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.890+0000 7f2a0f138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.890+0000 7f2a0f138640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.890+0000 7f2a0f138640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:36.894 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:36.890+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a08074cc0 con 0x7f2a081430c0 2024-03-25T18:24:37.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a00002060 con 0x7f2a081430c0 2024-03-25T18:24:37.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:37.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f2a00003b30 con 0x7f2a081430c0 2024-03-25T18:24:37.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:37.191 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a057fa640 10 monclient: dump: 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:37.192 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:37.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.190+0000 7f2a06ffd640 4 set_mon_vals no callback set 2024-03-25T18:24:37.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.191+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f2a00019b00 con 0x7f2a081430c0 2024-03-25T18:24:37.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.192+0000 7f2a0ced4640 10 monclient: get_auth_request con 0x7f29ec06e0d0 auth_method 0 2024-03-25T18:24:37.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.192+0000 7f2a057fa640 10 client.4721.objecter ms_handle_connect 0x7f29ec06e0d0 2024-03-25T18:24:37.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f2a00053880 con 0x7f2a081430c0 2024-03-25T18:24:37.198 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 10 client.4721.objecter ms_dispatch 0x7f2a08000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:37.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 3 client.4721.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:37.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 3 client.4721.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:37.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 20 client.4721.objecter dump_active .. 0 homeless 2024-03-25T18:24:37.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a0004f390 con 0x7f2a081430c0 2024-03-25T18:24:37.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:37.199 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.197+0000 7f2a057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:37.427 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.426+0000 7f2a0f138640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:37.428 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.426+0000 7f2a0f138640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:37.428 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.426+0000 7f2a0f138640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:37.428 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.426+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a080627b0 con 0x7f2a081430c0 2024-03-25T18:24:37.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.459+0000 7f2a057fa640 1 -- 172.21.15.132:0/3883751457 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f2a000512c0 con 0x7f2a081430c0 2024-03-25T18:24:37.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.459+0000 7f2a057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:37.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.459+0000 7f2a057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:37.460 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:37.460 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:37.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 10 librados: watch_flush enter 2024-03-25T18:24:37.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 10 librados: watch_flush exit 2024-03-25T18:24:37.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 20 client.4721.objecter shutdown clearing up homeless session... 2024-03-25T18:24:37.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 10 client.4721.objecter successfully canceled tick 2024-03-25T18:24:37.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 >> v1:172.21.15.161:6800/411950978 conn(0x7f29ec06e0d0 legacy=0x7f29ec070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.463 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 10 monclient: shutdown 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 >> v1:172.21.15.132:6790/0 conn(0x7f2a081430c0 legacy=0x7f2a08133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 shutdown_connections 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 >> 172.21.15.132:0/3883751457 conn(0x7f2a080b7ea0 msgr2=0x7f2a08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 shutdown_connections 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 -- 172.21.15.132:0/3883751457 wait complete. 2024-03-25T18:24:37.464 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:37.461+0000 7f2a0f138640 1 librados: shutdown 2024-03-25T18:24:37.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:44.280 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.391+0000 7f18814e6640 10 monclient: get_monmap_and_config 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.391+0000 7f18814e6640 10 monclient: build_initial_monmap 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.391+0000 7f18814e6640 10 monclient: monmap: 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:44.392930+0000 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:44.392930+0000 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:44.393 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:44.394 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:44.394 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.392+0000 7f18814e6640 1 Processor -- start 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.392+0000 7f18814e6640 1 -- start start 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.392+0000 7f18814e6640 10 monclient: init 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient(hunting): picked mon.noname-a con 0x7f187c0b4d60 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:44.395 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient(hunting): picked mon.noname-c con 0x7f187c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:44.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient(hunting): picked mon.noname-b con 0x7f187c14be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:44.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c0af940 con 0x7f187c0b4d60 2024-03-25T18:24:44.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c0b0a70 con 0x7f187c0b1730 2024-03-25T18:24:44.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c0b0d50 con 0x7f187c14be90 2024-03-25T18:24:44.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:44.396 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.393+0000 7f18814e6640 10 monclient(hunting): authenticate will time out at 7924.656531s 2024-03-25T18:24:44.401 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.400+0000 7f187affd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f187c0b4d60 0x7f187c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:49556/0 (socket says 172.21.15.132:49556) 2024-03-25T18:24:44.401 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.400+0000 7f187affd640 1 -- 172.21.15.132:0/3305689052 learned_addr learned my addr 172.21.15.132:0/3305689052 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:44.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.413+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074590309 0 0) 0x7f187c0b0a70 con 0x7f187c0b1730 2024-03-25T18:24:44.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.413+0000 7f187a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.413+0000 7f187a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.414 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.413+0000 7f187a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.413+0000 7f187a7fc640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:24:44.415 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.413+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1868003520 con 0x7f187c0b1730 2024-03-25T18:24:44.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.414+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144524399 0 0) 0x7f187c0af940 con 0x7f187c0b4d60 2024-03-25T18:24:44.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.414+0000 7f187a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.414+0000 7f187a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.414+0000 7f187a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.414+0000 7f187a7fc640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:24:44.416 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.415+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c0b0a70 con 0x7f187c0b4d60 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565166783 0 0) 0x7f187c0b0a70 con 0x7f187c0b4d60 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 >> v1:172.21.15.132:6790/0 conn(0x7f187c14be90 legacy=0x7f187c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 >> v1:172.21.15.161:6789/0 conn(0x7f187c0b1730 legacy=0x7f187c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c0b1030 con 0x7f187c0b4d60 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:44.424 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f186c001670 con 0x7f187c0b4d60 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f187a7fc640 10 monclient: dump: 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:44.425 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.426 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f18814e6640 5 monclient: authenticate success, global_id 4707 2024-03-25T18:24:44.426 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.422+0000 7f18814e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:44.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.428+0000 7f187a7fc640 1 -- 172.21.15.132:0/3305689052 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f186c0017d0 con 0x7f187c0b4d60 2024-03-25T18:24:44.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.428+0000 7f187a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.428+0000 7f18814e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:44.429 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.428+0000 7f18814e6640 4 set_mon_vals no callback set 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 10 monclient: shutdown 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 1 -- 172.21.15.132:0/3305689052 >> v1:172.21.15.132:6789/0 conn(0x7f187c0b4d60 legacy=0x7f187c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 1 -- 172.21.15.132:0/3305689052 shutdown_connections 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 1 -- 172.21.15.132:0/3305689052 >> 172.21.15.132:0/3305689052 conn(0x7f187c0617a0 msgr2=0x7f187c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 1 -- 172.21.15.132:0/3305689052 shutdown_connections 2024-03-25T18:24:44.430 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.429+0000 7f18814e6640 1 -- 172.21.15.132:0/3305689052 wait complete. 2024-03-25T18:24:44.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 10 monclient: build_initial_monmap 2024-03-25T18:24:44.431 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 10 monclient: monmap: 2024-03-25T18:24:44.431 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:44.431 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 1 librados: starting msgr at 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 1 librados: starting objecter 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 1 Processor -- start 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 1 -- start start 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 1 librados: setting wanted keys 2024-03-25T18:24:44.432 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 1 librados: calling monclient init 2024-03-25T18:24:44.433 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.430+0000 7f18814e6640 10 monclient: init 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient(hunting): picked mon.noname-b con 0x7f187c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient(hunting): picked mon.noname-c con 0x7f187c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient(hunting): picked mon.noname-a con 0x7f187c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c137d20 con 0x7f187c14be90 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c140a50 con 0x7f187c0b1730 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f187c138440 con 0x7f187c0b4d60 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:44.434 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.431+0000 7f18814e6640 10 monclient(hunting): authenticate will time out at 7924.694390s 2024-03-25T18:24:44.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.435+0000 7f1873fff640 1 --1- >> v1:172.21.15.132:6790/0 conn(0x7f187c0b4d60 0x7f187c1d7590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6790/0 says I am v1:172.21.15.132:43632/0 (socket says 172.21.15.132:43632) 2024-03-25T18:24:44.436 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.435+0000 7f1873fff640 1 -- 172.21.15.132:0/1677376241 learned_addr learned my addr 172.21.15.132:0/1677376241 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:44.437 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.437+0000 7f18737fe640 10 client.?.objecter ms_handle_connect 0x7f187c0b4d60 2024-03-25T18:24:44.438 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.437+0000 7f18737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.445+0000 7f18737fe640 10 client.?.objecter ms_handle_connect 0x7f187c14be90 2024-03-25T18:24:44.447 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.445+0000 7f18737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.451+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244178478 0 0) 0x7f187c138440 con 0x7f187c0b4d60 2024-03-25T18:24:44.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.451+0000 7f18737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.451+0000 7f18737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.451+0000 7f18737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.451+0000 7f18737fe640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:24:44.452 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.451+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c004eb0 con 0x7f187c0b4d60 2024-03-25T18:24:44.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.455+0000 7f18737fe640 10 client.?.objecter ms_handle_connect 0x7f187c0b1730 2024-03-25T18:24:44.457 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.456+0000 7f18737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:44.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.459+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483401246 0 0) 0x7f187c137d20 con 0x7f187c14be90 2024-03-25T18:24:44.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.459+0000 7f18737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.460 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.459+0000 7f18737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.459+0000 7f18737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.459+0000 7f18737fe640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:24:44.461 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.459+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c138440 con 0x7f187c14be90 2024-03-25T18:24:44.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.471+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447955184 0 0) 0x7f187c140a50 con 0x7f187c0b1730 2024-03-25T18:24:44.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.471+0000 7f18737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:44.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.471+0000 7f18737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:44.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.471+0000 7f18737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:44.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.471+0000 7f18737fe640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:24:44.472 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.471+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f187c137d20 con 0x7f187c0b1730 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1052413863 0 0) 0x7f185c004eb0 con 0x7f187c0b4d60 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f187c140a50 con 0x7f187c0b4d60 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1864001990 con 0x7f187c0b4d60 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 10 monclient(hunting): dump: 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.475 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.473+0000 7f18737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.484+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1190846140 0 0) 0x7f187c137d20 con 0x7f187c0b1730 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.484+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f185c004eb0 con 0x7f187c0b1730 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.484+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f186c0015b0 con 0x7f187c0b1730 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.484+0000 7f18737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.484+0000 7f18737fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.484+0000 7f18737fe640 10 monclient(hunting): dump: 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.488+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 285617514 0 0) 0x7f187c138440 con 0x7f187c14be90 2024-03-25T18:24:44.489 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.488+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f187c137d20 con 0x7f187c14be90 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.488+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1874001b50 con 0x7f187c14be90 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.488+0000 7f18737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.488+0000 7f18737fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.488+0000 7f18737fe640 10 monclient(hunting): dump: 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:44.490 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:44.491 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 812419049 0 0) 0x7f187c140a50 con 0x7f187c0b4d60 2024-03-25T18:24:44.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 >> v1:172.21.15.161:6789/0 conn(0x7f187c0b1730 legacy=0x7f187c131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.496 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 >> v1:172.21.15.132:6789/0 conn(0x7f187c14be90 legacy=0x7f187c1d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 1 monclient: found mon.c 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f187c1385d0 con 0x7f187c0b4d60 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 10 monclient: _renew_subs 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f187c144eb0 con 0x7f187c0b4d60 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:44.497 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 10 monclient: _renew_subs 2024-03-25T18:24:44.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:44.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f187c132420 con 0x7f187c0b4d60 2024-03-25T18:24:44.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.495+0000 7f18814e6640 1 librados: init done 2024-03-25T18:24:44.498 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.496+0000 7f18814e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.496+0000 7f18814e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.496+0000 7f18814e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:44.499 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.496+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f187c0627b0 con 0x7f187c0b4d60 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1864003610 con 0x7f187c0b4d60 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f18737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f1864003960 con 0x7f187c0b4d60 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f18737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f18737fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f18737fe640 10 monclient: dump: 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:44.521 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:44.522 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.520+0000 7f1879ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:44.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.817+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f1864003ac0 con 0x7f187c0b4d60 2024-03-25T18:24:44.818 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.817+0000 7f187affd640 10 monclient: get_auth_request con 0x7f185c06e0f0 auth_method 0 2024-03-25T18:24:44.819 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.818+0000 7f18737fe640 10 client.4727.objecter ms_handle_connect 0x7f185c06e0f0 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f1864054710 con 0x7f187c0b4d60 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 10 client.4727.objecter ms_dispatch 0x7f187c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1864050220 con 0x7f187c0b4d60 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.829 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:44.828+0000 7f18737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:45.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.078+0000 7f18814e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:45.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.078+0000 7f18814e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:45.079 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.078+0000 7f18814e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:45.080 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.078+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f187c140be0 con 0x7f187c0b4d60 2024-03-25T18:24:45.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.117+0000 7f18737fe640 1 -- 172.21.15.132:0/1677376241 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f1864052150 con 0x7f187c0b4d60 2024-03-25T18:24:45.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.117+0000 7f18737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:45.118 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.117+0000 7f18737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:45.118 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:45.118 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 10 librados: watch_flush enter 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 10 librados: watch_flush exit 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 10 client.4727.objecter successfully canceled tick 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 >> v1:172.21.15.161:6800/411950978 conn(0x7f185c06e0f0 legacy=0x7f185c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 10 monclient: shutdown 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 >> v1:172.21.15.132:6790/0 conn(0x7f187c0b4d60 legacy=0x7f187c1d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 shutdown_connections 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 >> 172.21.15.132:0/1677376241 conn(0x7f187c0617a0 msgr2=0x7f187c0b6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 shutdown_connections 2024-03-25T18:24:45.121 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 -- 172.21.15.132:0/1677376241 wait complete. 2024-03-25T18:24:45.122 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:45.119+0000 7f18814e6640 1 librados: shutdown 2024-03-25T18:24:45.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:51.939 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.058+0000 7fa930f89640 10 monclient: get_monmap_and_config 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.058+0000 7fa930f89640 10 monclient: build_initial_monmap 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.058+0000 7fa930f89640 10 monclient: monmap: 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:52.059167+0000 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:52.059167+0000 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:52.059 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:52.060 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:52.060 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:52.060 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.058+0000 7fa930f89640 1 Processor -- start 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.058+0000 7fa930f89640 1 -- start start 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.058+0000 7fa930f89640 10 monclient: init 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient(hunting): picked mon.noname-b con 0x7fa92c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient(hunting): picked mon.noname-c con 0x7fa92c0b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:52.061 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient(hunting): picked mon.noname-a con 0x7fa92c14be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:52.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0af940 con 0x7fa92c14be90 2024-03-25T18:24:52.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0b0a70 con 0x7fa92c0b1730 2024-03-25T18:24:52.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c0b0d50 con 0x7fa92c0b4d60 2024-03-25T18:24:52.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:52.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.059+0000 7fa930f89640 10 monclient(hunting): authenticate will time out at 7932.322075s 2024-03-25T18:24:52.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.071+0000 7fa92ad9d640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa92c14be90 0x7fa92c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54534/0 (socket says 172.21.15.132:54534) 2024-03-25T18:24:52.072 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.071+0000 7fa92ad9d640 1 -- 172.21.15.132:0/1527396325 learned_addr learned my addr 172.21.15.132:0/1527396325 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:52.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.088+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651060136 0 0) 0x7fa92c0b0a70 con 0x7fa92c0b1730 2024-03-25T18:24:52.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.088+0000 7fa929d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.088+0000 7fa929d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.088+0000 7fa929d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.088+0000 7fa929d9b640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:24:52.089 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.088+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa918003520 con 0x7fa92c0b1730 2024-03-25T18:24:52.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.091+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261673239 0 0) 0x7fa92c0af940 con 0x7fa92c14be90 2024-03-25T18:24:52.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.091+0000 7fa929d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.091+0000 7fa929d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.091+0000 7fa929d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.091+0000 7fa929d9b640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:24:52.093 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.091+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c0b0a70 con 0x7fa92c14be90 2024-03-25T18:24:52.107 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.106+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995818290 0 0) 0x7fa92c0b0d50 con 0x7fa92c0b4d60 2024-03-25T18:24:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.106+0000 7fa929d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.106+0000 7fa929d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.106+0000 7fa929d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.106+0000 7fa929d9b640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:24:52.108 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.106+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c0af940 con 0x7fa92c0b4d60 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3650532374 0 0) 0x7fa918003520 con 0x7fa92c0b1730 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 >> v1:172.21.15.132:6790/0 conn(0x7fa92c0b4d60 legacy=0x7fa92c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 >> v1:172.21.15.132:6789/0 conn(0x7fa92c14be90 legacy=0x7fa92c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 1 monclient: found mon.noname-c 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.161:6789/0 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0b1030 con 0x7fa92c0b1730 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:52.110 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa914001690 con 0x7fa92c0b1730 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa930f89640 5 monclient: authenticate success, global_id 4738 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa930f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa929d9b640 10 monclient: dump: 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:52.111 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:52.112 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:52.112 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.112 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.108+0000 7fa930f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.1 v1:172.21.15.161:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9140017f0 con 0x7fa92c0b1730 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa929d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa929d9b640 1 -- 172.21.15.132:0/1527396325 <== mon.1 v1:172.21.15.161:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa914001c00 con 0x7fa92c0b1730 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa930f89640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa930f89640 4 set_mon_vals no callback set 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa929d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa929d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa929d9b640 10 monclient: dump: 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:52.131 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:52.132 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa930f89640 10 monclient: shutdown 2024-03-25T18:24:52.133 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa930f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.130+0000 7fa930f89640 1 -- 172.21.15.132:0/1527396325 >> v1:172.21.15.161:6789/0 conn(0x7fa92c0b1730 legacy=0x7fa92c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.131+0000 7fa930f89640 1 -- 172.21.15.132:0/1527396325 shutdown_connections 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.131+0000 7fa930f89640 1 -- 172.21.15.132:0/1527396325 >> 172.21.15.132:0/1527396325 conn(0x7fa92c0617a0 msgr2=0x7fa92c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.131+0000 7fa930f89640 1 -- 172.21.15.132:0/1527396325 shutdown_connections 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.131+0000 7fa930f89640 1 -- 172.21.15.132:0/1527396325 wait complete. 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 10 monclient: build_initial_monmap 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 10 monclient: monmap: 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 1 librados: starting msgr at 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 1 librados: starting objecter 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 1 Processor -- start 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 1 -- start start 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 1 librados: setting wanted keys 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 1 librados: calling monclient init 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.132+0000 7fa930f89640 10 monclient: init 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient(hunting): picked mon.noname-a con 0x7fa92c0b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient(hunting): picked mon.noname-c con 0x7fa92c0b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient(hunting): picked mon.noname-b con 0x7fa92c14be90 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c135810 con 0x7fa92c0b1730 2024-03-25T18:24:52.136 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c144c10 con 0x7fa92c14be90 2024-03-25T18:24:52.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c144ef0 con 0x7fa92c0b4d60 2024-03-25T18:24:52.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:52.137 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.133+0000 7fa930f89640 10 monclient(hunting): authenticate will time out at 7932.396493s 2024-03-25T18:24:52.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.145+0000 7fa92a59c640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7fa92c0b1730 0x7fa92c1d7980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:54580/0 (socket says 172.21.15.132:54580) 2024-03-25T18:24:52.146 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.145+0000 7fa92a59c640 1 -- 172.21.15.132:0/2525606995 learned_addr learned my addr 172.21.15.132:0/2525606995 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:52.148 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.147+0000 7fa9237fe640 10 client.?.objecter ms_handle_connect 0x7fa92c0b4d60 2024-03-25T18:24:52.149 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.147+0000 7fa9237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:52.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.149+0000 7fa9237fe640 10 client.?.objecter ms_handle_connect 0x7fa92c0b1730 2024-03-25T18:24:52.150 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.149+0000 7fa9237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:52.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.154+0000 7fa9237fe640 10 client.?.objecter ms_handle_connect 0x7fa92c14be90 2024-03-25T18:24:52.155 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.154+0000 7fa9237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:52.161 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.160+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941939556 0 0) 0x7fa92c144c10 con 0x7fa92c14be90 2024-03-25T18:24:52.162 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.160+0000 7fa9237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.162 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.160+0000 7fa9237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.162 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.160+0000 7fa9237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.162 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.160+0000 7fa9237fe640 10 monclient(hunting): my global_id is 4741 2024-03-25T18:24:52.162 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.161+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa92c14be90 2024-03-25T18:24:52.167 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.166+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141359578 0 0) 0x7fa92c135810 con 0x7fa92c0b1730 2024-03-25T18:24:52.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.166+0000 7fa9237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.166+0000 7fa9237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.166+0000 7fa9237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.166+0000 7fa9237fe640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:24:52.168 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.166+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c144c10 con 0x7fa92c0b1730 2024-03-25T18:24:52.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.167+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042811792 0 0) 0x7fa92c144ef0 con 0x7fa92c0b4d60 2024-03-25T18:24:52.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.168+0000 7fa9237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:52.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.168+0000 7fa9237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:52.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.168+0000 7fa9237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:52.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.168+0000 7fa9237fe640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:24:52.169 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.168+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa92c135810 con 0x7fa92c0b4d60 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588538811 0 0) 0x7fa90c004eb0 con 0x7fa92c14be90 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa92c144ef0 con 0x7fa92c14be90 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa924001bd0 con 0x7fa92c14be90 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 10 monclient(hunting): dump: 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:52.185 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.186 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.183+0000 7fa9237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.192+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2286596311 0 0) 0x7fa92c144c10 con 0x7fa92c0b1730 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.192+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.132:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa90c004eb0 con 0x7fa92c0b1730 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.192+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa91c0019f0 con 0x7fa92c0b1730 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.192+0000 7fa9237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.192+0000 7fa9237fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.192+0000 7fa9237fe640 10 monclient(hunting): dump: 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:52.193 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:52.194 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2480192953 0 0) 0x7fa92c144ef0 con 0x7fa92c14be90 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 >> v1:172.21.15.132:6790/0 conn(0x7fa92c0b4d60 legacy=0x7fa92c1349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 >> v1:172.21.15.132:6789/0 conn(0x7fa92c0b1730 legacy=0x7fa92c1d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 1 monclient: found mon.b 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c140c10 con 0x7fa92c14be90 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa9237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 5 monclient: authenticate success, global_id 4741 2024-03-25T18:24:52.201 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 10 monclient: _renew_subs 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92c137d50 con 0x7fa92c14be90 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 10 client.4741.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 10 monclient: _renew_subs 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92c1407d0 con 0x7fa92c14be90 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.199+0000 7fa930f89640 1 librados: init done 2024-03-25T18:24:52.202 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.201+0000 7fa930f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:52.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.201+0000 7fa930f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:52.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.201+0000 7fa930f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:52.203 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.201+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92c141e30 con 0x7fa92c14be90 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa924004b60 con 0x7fa92c14be90 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa9237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7fa924002100 con 0x7fa92c14be90 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa9237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa9237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa9237fe640 10 monclient: dump: 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:52.476 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.474+0000 7fa92959a640 4 set_mon_vals no callback set 2024-03-25T18:24:52.477 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.475+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7fa92401ad20 con 0x7fa92c14be90 2024-03-25T18:24:52.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.476+0000 7fa92a59c640 10 monclient: get_auth_request con 0x7fa90c06ded0 auth_method 0 2024-03-25T18:24:52.478 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.476+0000 7fa9237fe640 10 client.4741.objecter ms_handle_connect 0x7fa90c06ded0 2024-03-25T18:24:52.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.480+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7fa924054a10 con 0x7fa92c14be90 2024-03-25T18:24:52.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.480+0000 7fa9237fe640 10 client.4741.objecter ms_dispatch 0x7fa92c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:52.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.480+0000 7fa9237fe640 3 client.4741.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:52.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.480+0000 7fa9237fe640 3 client.4741.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:52.481 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.480+0000 7fa9237fe640 20 client.4741.objecter dump_active .. 0 homeless 2024-03-25T18:24:52.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.481+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa924050350 con 0x7fa92c14be90 2024-03-25T18:24:52.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.481+0000 7fa9237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:52.482 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.481+0000 7fa9237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:52.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.701+0000 7fa930f89640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.701+0000 7fa930f89640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.701+0000 7fa930f89640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.161:6789/0 2024-03-25T18:24:52.702 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.701+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 --> v1:172.21.15.161:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa92c1380a0 con 0x7fa92c14be90 2024-03-25T18:24:52.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.728+0000 7fa9237fe640 1 -- 172.21.15.132:0/2525606995 <== mon.1 v1:172.21.15.161:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7fa92c1380a0 con 0x7fa92c14be90 2024-03-25T18:24:52.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.728+0000 7fa9237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.729 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.728+0000 7fa9237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:52.729 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:24:52.729 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 10 librados: watch_flush enter 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 10 librados: watch_flush exit 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 20 client.4741.objecter shutdown clearing up homeless session... 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 10 client.4741.objecter successfully canceled tick 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 >> v1:172.21.15.161:6800/411950978 conn(0x7fa90c06ded0 legacy=0x7fa90c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 10 monclient: shutdown 2024-03-25T18:24:52.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:52.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 >> v1:172.21.15.161:6789/0 conn(0x7fa92c14be90 legacy=0x7fa92c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 shutdown_connections 2024-03-25T18:24:52.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 >> 172.21.15.132:0/2525606995 conn(0x7fa92c0617a0 msgr2=0x7fa92c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:52.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 shutdown_connections 2024-03-25T18:24:52.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.730+0000 7fa930f89640 1 -- 172.21.15.132:0/2525606995 wait complete. 2024-03-25T18:24:52.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:52.731+0000 7fa930f89640 1 librados: shutdown 2024-03-25T18:24:52.745 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:56.124 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:41.388 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.132 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:41.396 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.158 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:11:41.422 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.185 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:41.449 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.188 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:41.429 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.192 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:41.456 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.198 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:41.439 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.217 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:11:41.481 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:56.221 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:11:41.463 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.552 DEBUG:teuthology.orchestra.run.smithi132:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.667+0000 7f986ddb5640 10 monclient: get_monmap_and_config 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.667+0000 7f986ddb5640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.667+0000 7f986ddb5640 10 monclient: monmap: 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:24:59.668185+0000 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:24:59.668185+0000 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.668 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:59.669 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:59.669 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-c 2024-03-25T18:24:59.669 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-b 2024-03-25T18:24:59.669 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 1 Processor -- start 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 1 -- start start 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient: init 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient(hunting): picked mon.noname-c con 0x7f98680b4d60 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient(hunting): picked mon.noname-a con 0x7f98680b1730 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient(hunting): picked mon.noname-b con 0x7f986814be90 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:59.670 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680af940 con 0x7f98680b1730 2024-03-25T18:24:59.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680b0a70 con 0x7f98680b4d60 2024-03-25T18:24:59.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98680b0d50 con 0x7f986814be90 2024-03-25T18:24:59.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.671 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.668+0000 7f986ddb5640 10 monclient(hunting): authenticate will time out at 7939.931415s 2024-03-25T18:24:59.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.673+0000 7f9866ffd640 1 --1- >> v1:172.21.15.132:6789/0 conn(0x7f98680b1730 0x7f98680b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.132:6789/0 says I am v1:172.21.15.132:33748/0 (socket says 172.21.15.132:33748) 2024-03-25T18:24:59.674 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.673+0000 7f9866ffd640 1 -- 172.21.15.132:0/1182553075 learned_addr learned my addr 172.21.15.132:0/1182553075 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:59.684 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.683+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102957694 0 0) 0x7f98680af940 con 0x7f98680b1730 2024-03-25T18:24:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.683+0000 7f98667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.683+0000 7f98667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.683+0000 7f98667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.683+0000 7f98667fc640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:24:59.685 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.683+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9850003520 con 0x7f98680b1730 2024-03-25T18:24:59.698 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 <== mon.0 v1:172.21.15.132:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2908669690 0 0) 0x7f9850003520 con 0x7f98680b1730 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 >> v1:172.21.15.132:6790/0 conn(0x7f986814be90 legacy=0x7f986814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 >> v1:172.21.15.161:6789/0 conn(0x7f98680b4d60 legacy=0x7f986813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 1 monclient: found mon.noname-a 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.132:6789/0 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 --> v1:172.21.15.132:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98680b1030 con 0x7f98680b1730 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 <== mon.0 v1:172.21.15.132:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f985c001690 con 0x7f98680b1730 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f98667fc640 10 monclient: dump: 2024-03-25T18:24:59.699 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f986ddb5640 5 monclient: authenticate success, global_id 4719 2024-03-25T18:24:59.700 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.697+0000 7f986ddb5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:59.713 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 <== mon.0 v1:172.21.15.132:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f985c001860 con 0x7f98680b1730 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f98667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f98667fc640 1 -- 172.21.15.132:0/1182553075 <== mon.0 v1:172.21.15.132:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f985c001cb0 con 0x7f98680b1730 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f98667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f98667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f98667fc640 10 monclient: dump: 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:59.714 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:59.715 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:59.715 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:59.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f986ddb5640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:59.715 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.712+0000 7f986ddb5640 4 set_mon_vals no callback set 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 10 monclient: shutdown 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 1 -- 172.21.15.132:0/1182553075 >> v1:172.21.15.132:6789/0 conn(0x7f98680b1730 legacy=0x7f98680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 1 -- 172.21.15.132:0/1182553075 shutdown_connections 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 1 -- 172.21.15.132:0/1182553075 >> 172.21.15.132:0/1182553075 conn(0x7f98680617a0 msgr2=0x7f9868061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 1 -- 172.21.15.132:0/1182553075 shutdown_connections 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.713+0000 7f986ddb5640 1 -- 172.21.15.132:0/1182553075 wait complete. 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 10 monclient: monmap: 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:epoch 0 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:59.717 INFO:teuthology.misc.health.smithi132.stderr:last_changed 0.000000 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:created 0.000000 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 1 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.noname-a 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.noname-b 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.noname-c 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 1 librados: starting msgr at 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 1 librados: starting objecter 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 1 Processor -- start 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 1 -- start start 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 1 librados: setting wanted keys 2024-03-25T18:24:59.718 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 1 librados: calling monclient init 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.714+0000 7f986ddb5640 10 monclient: init 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient(hunting): picked mon.noname-b con 0x7f98680b1730 addr v1:172.21.15.161:6789/0 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient(hunting): picked mon.noname-c con 0x7f98680b4d60 addr v1:172.21.15.132:6790/0 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient(hunting): picked mon.noname-a con 0x7f986814be90 addr v1:172.21.15.132:6789/0 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 1 -- --> v1:172.21.15.132:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986813ac40 con 0x7f986814be90 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98681398e0 con 0x7f98680b1730 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 1 -- --> v1:172.21.15.132:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f986813ae90 con 0x7f98680b4d60 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.719 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.715+0000 7f986ddb5640 10 monclient(hunting): authenticate will time out at 7939.978128s 2024-03-25T18:24:59.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.719+0000 7f98677fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f98680b1730 0x7f98681391d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.132:58106/0 (socket says 172.21.15.132:58106) 2024-03-25T18:24:59.720 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.719+0000 7f98677fe640 1 -- 172.21.15.132:0/1115812782 learned_addr learned my addr 172.21.15.132:0/1115812782 (peer_addr_for_me v1:172.21.15.132:0/0) 2024-03-25T18:24:59.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.720+0000 7f984ffff640 10 client.?.objecter ms_handle_connect 0x7f98680b1730 2024-03-25T18:24:59.721 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.720+0000 7f984ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.723+0000 7f984ffff640 10 client.?.objecter ms_handle_connect 0x7f98680b4d60 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.723+0000 7f984ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.727+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.1 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825817024 0 0) 0x7f98681398e0 con 0x7f98680b1730 2024-03-25T18:24:59.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.727+0000 7f984ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.727+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.727+0000 7f984ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.727+0000 7f984ffff640 10 monclient(hunting): my global_id is 4744 2024-03-25T18:24:59.728 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.727+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9854004eb0 con 0x7f98680b1730 2024-03-25T18:24:59.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.731+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741864039 0 0) 0x7f986813ae90 con 0x7f98680b4d60 2024-03-25T18:24:59.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.731+0000 7f984ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.732 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.731+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.731+0000 7f984ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.731+0000 7f984ffff640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:24:59.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.731+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98681398e0 con 0x7f98680b4d60 2024-03-25T18:24:59.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.732+0000 7f984ffff640 10 client.?.objecter ms_handle_connect 0x7f986814be90 2024-03-25T18:24:59.733 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.732+0000 7f984ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1296407505 0 0) 0x7f98681398e0 con 0x7f98680b4d60 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f986813ae90 con 0x7f98680b4d60 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f985c001b70 con 0x7f98680b4d60 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 10 monclient(hunting): dump: 2024-03-25T18:24:59.742 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:24:59.743 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.741+0000 7f984ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:59.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.742+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.0 v1:172.21.15.132:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960909904 0 0) 0x7f986813ac40 con 0x7f986814be90 2024-03-25T18:24:59.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.742+0000 7f984ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.742+0000 7f984ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.742+0000 7f984ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.742+0000 7f984ffff640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:24:59.744 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.742+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98681398e0 con 0x7f986814be90 2024-03-25T18:24:59.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 133848978 0 0) 0x7f986813ae90 con 0x7f98680b4d60 2024-03-25T18:24:59.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 >> v1:172.21.15.161:6789/0 conn(0x7f98680b1730 legacy=0x7f98681391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.750 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 >> v1:172.21.15.132:6789/0 conn(0x7f986814be90 legacy=0x7f98681d7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 1 monclient: found mon.c 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98681daf90 con 0x7f98680b4d60 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 10 monclient: _finish_auth 0 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f984ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 5 monclient: authenticate success, global_id 4739 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 10 monclient: _renew_subs 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98681d79f0 con 0x7f98680b4d60 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 10 client.4739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 10 monclient: _renew_subs 2024-03-25T18:24:59.751 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:59.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98681dbda0 con 0x7f98680b4d60 2024-03-25T18:24:59.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.749+0000 7f986ddb5640 1 librados: init done 2024-03-25T18:24:59.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.750+0000 7f986ddb5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.750+0000 7f986ddb5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.750+0000 7f986ddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:24:59.752 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:24:59.750+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98680627b0 con 0x7f98680b4d60 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f985c001cd0 con 0x7f98680b4d60 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f984ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1548684331 0 0) 0x7f985c001250 con 0x7f98680b4d60 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f984ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f984ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f984ffff640 10 monclient: dump: 2024-03-25T18:25:00.054 INFO:teuthology.misc.health.smithi132.stderr:epoch 1 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:fsid 6e953815-4f55-4f47-8818-b2f2072285a9 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:last_changed 2024-03-25T18:12:41.831165+0000 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:created 2024-03-25T18:12:41.831165+0000 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:election_strategy: 3 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:0: v1:172.21.15.132:6789/0 mon.a 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:1: v1:172.21.15.161:6789/0 mon.b 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:2: v1:172.21.15.132:6790/0 mon.c 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr: 2024-03-25T18:25:00.055 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.053+0000 7f9865ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:00.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.054+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4087179551 0 0) 0x7f985c01ac00 con 0x7f98680b4d60 2024-03-25T18:25:00.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.054+0000 7f98677fe640 10 monclient: get_auth_request con 0x7f985406ddc0 auth_method 0 2024-03-25T18:25:00.056 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.055+0000 7f984ffff640 10 client.4739.objecter ms_handle_connect 0x7f985406ddc0 2024-03-25T18:25:00.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3442129276 0 0) 0x7f985c054820 con 0x7f98680b4d60 2024-03-25T18:25:00.062 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 10 client.4739.objecter ms_dispatch 0x7f9868000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:00.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 3 client.4739.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:00.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 3 client.4739.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:00.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 20 client.4739.objecter dump_active .. 0 homeless 2024-03-25T18:25:00.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f985c050160 con 0x7f98680b4d60 2024-03-25T18:25:00.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:00.063 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.061+0000 7f984ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:00.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.307+0000 7f986ddb5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:00.308 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.307+0000 7f986ddb5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:00.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.307+0000 7f986ddb5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.132:6790/0 2024-03-25T18:25:00.309 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.307+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 --> v1:172.21.15.132:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98681db120 con 0x7f98680b4d60 2024-03-25T18:25:00.339 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.338+0000 7f984ffff640 1 -- 172.21.15.132:0/1115812782 <== mon.2 v1:172.21.15.132:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2675919931) 0x7f985c052260 con 0x7f98680b4d60 2024-03-25T18:25:00.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.338+0000 7f984ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:00.340 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.338+0000 7f984ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:00.340 INFO:teuthology.misc.health.smithi132.stdout: 2024-03-25T18:25:00.340 INFO:teuthology.misc.health.smithi132.stdout:{"epoch":11,"fsid":"6e953815-4f55-4f47-8818-b2f2072285a9","created":"2024-03-25T18:13:30.955024+0000","modified":"2024-03-25T18:23:35.778222+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:23:35.778222+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9a272c8b-fb39-4075-b438-a2a51b258a70","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"eff0a134-4085-4f7d-840d-fa3e3b399d35","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f38d64d8-cdc7-4da6-9196-a7ebbcd8fad8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"55459360-8371-4a31-91da-0b4dcb0523f0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dd421c87-126d-478b-b501-c1b526011ab9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"d0a69e52-0ecc-4545-a092-69ba9d72c8be","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4b872c23-c963-4cd7-89c0-336369dc9303","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b0ef21de-3118-43fd-9712-52f4834fd5a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:00.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 10 librados: watch_flush enter 2024-03-25T18:25:00.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 10 librados: watch_flush exit 2024-03-25T18:25:00.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 20 client.4739.objecter shutdown clearing up homeless session... 2024-03-25T18:25:00.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 10 client.4739.objecter successfully canceled tick 2024-03-25T18:25:00.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 >> v1:172.21.15.161:6800/411950978 conn(0x7f985406ddc0 legacy=0x7f9854070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.343 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 10 monclient: shutdown 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 >> v1:172.21.15.132:6790/0 conn(0x7f98680b4d60 legacy=0x7f98681db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 shutdown_connections 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.341+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 >> 172.21.15.132:0/1115812782 conn(0x7f98680617a0 msgr2=0x7f9868137370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.342+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 shutdown_connections 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.342+0000 7f986ddb5640 1 -- 172.21.15.132:0/1115812782 wait complete. 2024-03-25T18:25:00.344 INFO:teuthology.misc.health.smithi132.stderr:2024-03-25T18:25:00.342+0000 7f986ddb5640 1 librados: shutdown 2024-03-25T18:25:00.356 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:00.356 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:25:00.357 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:25:00.358 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:25:00.358 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:02.256 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:11:24.742 35202== 2024-03-25T18:25:02.257 INFO:tasks.ceph.osd.0.smithi132.stderr:==00:00:11:24.742 35202== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:02.301 INFO:tasks.ceph.osd.0.smithi132.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:06.459 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:25:06.460 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:25:06.461 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:25:06.461 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:08.499 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:11:30.965 35204== 2024-03-25T18:25:08.499 INFO:tasks.ceph.osd.1.smithi132.stderr:==00:00:11:30.965 35204== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:08.539 INFO:tasks.ceph.osd.1.smithi132.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:12.562 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:25:12.563 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:25:12.563 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:25:12.563 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:14.402 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:11:36.884 35203== 2024-03-25T18:25:14.402 INFO:tasks.ceph.osd.2.smithi132.stderr:==00:00:11:36.884 35203== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:14.439 INFO:tasks.ceph.osd.2.smithi132.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:16.258 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:01.522 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.312 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:01.576 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.338 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:01.602 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.353 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:01.617 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.363 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:01.627 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.365 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:01.606 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.373 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:01.615 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.395 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:01.637 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.400 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:01.664 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:18.665 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:25:18.665 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:25:18.666 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:25:18.666 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:20.630 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:11:43.096 35205== 2024-03-25T18:25:20.630 INFO:tasks.ceph.osd.3.smithi132.stderr:==00:00:11:43.096 35205== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:20.670 INFO:tasks.ceph.osd.3.smithi132.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:24.768 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:25:24.768 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:25:24.768 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:25:24.769 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:26.855 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:11:49.340 35307== 2024-03-25T18:25:26.855 INFO:tasks.ceph.osd.4.smithi161.stderr:==00:00:11:49.340 35307== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:26.898 INFO:tasks.ceph.osd.4.smithi161.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:30.870 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:25:30.871 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:25:30.871 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:25:30.871 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:32.684 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:11:55.164 35308== 2024-03-25T18:25:32.684 INFO:tasks.ceph.osd.5.smithi161.stderr:==00:00:11:55.164 35308== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:32.722 INFO:tasks.ceph.osd.5.smithi161.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:36.239 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:21.503 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.246 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:21.510 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.266 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:21.529 34890== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.289 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:21.553 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.295 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:21.537 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.296 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:21.560 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.299 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:21.540 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.320 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:21.584 34887== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.321 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:21.562 34685== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:36.973 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:25:36.973 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:25:36.974 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:25:36.974 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:38.678 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:12:01.157 35309== 2024-03-25T18:25:38.679 INFO:tasks.ceph.osd.6.smithi161.stderr:==00:00:12:01.157 35309== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:38.716 INFO:tasks.ceph.osd.6.smithi161.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:43.075 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:43.076 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:25:43.076 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:25:43.076 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:25:43.076 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:44.520 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:12:06.998 35310== 2024-03-25T18:25:44.520 INFO:tasks.ceph.osd.7.smithi161.stderr:==00:00:12:06.998 35310== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:44.553 INFO:tasks.ceph.osd.7.smithi161.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:49.179 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:25:49.179 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:25:49.180 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:25:49.180 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:25:49.180 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:49.205 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:25:49.205 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:25:49.205 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:49.206 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:25:49.205+0000 7f29fdb51640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34683) UID: 0 2024-03-25T18:25:49.206 INFO:tasks.ceph.mgr.x.smithi161.stderr:2024-03-25T18:25:49.205+0000 7f29fdb51640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:25:49.234 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:25:49.234 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:25:49.234 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:25:49.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:49.262 INFO:tasks.ceph.mon.a.smithi132.stderr:2024-03-25T18:25:49.250+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34856) UID: 0 2024-03-25T18:25:49.262 INFO:tasks.ceph.mon.a.smithi132.stderr:2024-03-25T18:25:49.251+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:25:49.690 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:34.954 34887== 2024-03-25T18:25:49.690 INFO:tasks.ceph.mon.a.smithi132.stderr:==00:00:12:34.954 34887== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:49.699 INFO:tasks.ceph.mon.a.smithi132.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:55.336 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:25:55.336 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:25:55.337 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:25:55.337 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:55.354 INFO:tasks.ceph.mon.c.smithi132.stderr:2024-03-25T18:25:55.340+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34858) UID: 0 2024-03-25T18:25:55.355 INFO:tasks.ceph.mon.c.smithi132.stderr:2024-03-25T18:25:55.342+0000 9f86640 -1 mon.c@2(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:25:55.927 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:41.191 34890== 2024-03-25T18:25:55.927 INFO:tasks.ceph.mon.c.smithi132.stderr:==00:00:12:41.191 34890== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:55.937 INFO:tasks.ceph.mon.c.smithi132.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:01.438 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi132 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:26:01.438 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:26:01.439 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:26:01.439 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:01.454 INFO:tasks.ceph.mon.b.smithi161.stderr:2024-03-25T18:26:01.441+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34681) UID: 0 2024-03-25T18:26:01.454 INFO:tasks.ceph.mon.b.smithi161.stderr:2024-03-25T18:26:01.443+0000 9f86640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:26:02.080 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:47.321 34685== 2024-03-25T18:26:02.080 INFO:tasks.ceph.mon.b.smithi161.stderr:==00:00:12:47.321 34685== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:02.089 INFO:tasks.ceph.mon.b.smithi161.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:07.541 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:26:07.541 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:26:07.542 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:26:07.542 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:26:07.661 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:26:07.661 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:26:07.809 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:26:07.809 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:26:07.915 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:26:07.915 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:26:07.981 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:26:07.982 DEBUG:teuthology.orchestra.run.smithi132:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:26:08.047 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:26:08.048 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:08.222 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:26:08.222 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:08.301 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:26:08.302 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:08.359 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:26:08.359 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:08.425 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:26:08.427 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/data/mon.a.tgz 2024-03-25T18:26:08.428 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-25T18:26:08.444 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.uSxBdjlLOK 2024-03-25T18:26:08.444 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.uSxBdjlLOK 2024-03-25T18:26:08.546 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.uSxBdjlLOK 2024-03-25T18:26:08.630 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.uSxBdjlLOK is 134KB 2024-03-25T18:26:08.682 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.uSxBdjlLOK 2024-03-25T18:26:08.696 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/data/mon.c.tgz 2024-03-25T18:26:08.696 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2024-03-25T18:26:08.750 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.HRVetAUud8 2024-03-25T18:26:08.751 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.HRVetAUud8 2024-03-25T18:26:08.839 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.HRVetAUud8 2024-03-25T18:26:08.923 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.HRVetAUud8 is 134KB 2024-03-25T18:26:08.975 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.HRVetAUud8 2024-03-25T18:26:08.989 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/data/mon.b.tgz 2024-03-25T18:26:08.989 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-03-25T18:26:09.006 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.PECHIglhau 2024-03-25T18:26:09.006 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.PECHIglhau 2024-03-25T18:26:09.107 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.PECHIglhau 2024-03-25T18:26:09.194 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.PECHIglhau is 134KB 2024-03-25T18:26:09.246 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.PECHIglhau 2024-03-25T18:26:09.261 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:26:09.262 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:26:09.264 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:26:09.617 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:09.887 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:26:09.889 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/remote/smithi132/crash 2024-03-25T18:26:09.890 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:26:09.917 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/remote/smithi161/crash 2024-03-25T18:26:09.919 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:26:09.945 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:26:09.946 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:26:09.962 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35048.log 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35130.log 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.35048.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.35130.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35048.log.gz 2024-03-25T18:26:10.033 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/valgrind/osd.7.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35130.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/valgrind/osd.5.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/valgrind/osd.6.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr: 72.5% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34839.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34879.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.34839.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.34879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34999.log 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.34839.log.gz 2024-03-25T18:26:10.034 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34879.log.gz 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.34999.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34999.log.gz 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34919.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35090.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.34919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34959.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.35090.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34919.log.gz 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.34959.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.35090.log.gz 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:26:10.035 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34959.log.gz/var/log/ceph/ceph-mgr.x.log: 2024-03-25T18:26:10.036 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40144.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41082.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40144.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40144.log.gz 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39166.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41082.log.gz 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40026.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38971.log 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2024-03-25T18:26:10.040 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38815.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39166.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36296.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40026.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38815.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38971.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.36296.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38815.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40417.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36296.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:26:10.041 INFO:teuthology.orchestra.run.smithi132.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41043.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40496.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39870.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40692.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41043.log.gz 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40887.log 2024-03-25T18:26:10.042 INFO:teuthology.orchestra.run.smithi132.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40496.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40692.log: /var/log/ceph/ceph-client.admin.39283.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39870.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41902.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40692.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41550.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40887.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41902.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39283.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41238.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40339.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41902.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40536.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41550.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39322.log 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41238.log.gz 2024-03-25T18:26:10.043 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose 86.9%/var/log/ceph/ceph-client.admin.40536.log: -- -- replaced with /var/log/ceph/ceph-client.admin.40339.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: /var/log/ceph/ceph-client.admin.40731.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40536.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40926.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39322.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.35390.log: gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40731.log.gz -5 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41589.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40926.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29045.log 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41589.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39675.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-03-25T18:26:10.044 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41277.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.tmp-client.admin.29045.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29045.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39636.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42019.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41277.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39363.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.42019.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39636.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41707.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41199.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42019.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39792.log 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39363.log.gz 2024-03-25T18:26:10.045 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40261.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41707.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40300.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41199.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40770.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39792.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40261.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40809.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40300.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40770.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39244.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39753.log 2024-03-25T18:26:10.046 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40809.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39714.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39753.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39244.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40653.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39753.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40222.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39714.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39480.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41980.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40653.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41668.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40222.log.gz 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39987.log 2024-03-25T18:26:10.047 INFO:teuthology.orchestra.run.smithi132.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39480.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39519.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41668.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41980.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39402.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39519.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41668.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41472.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39987.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39831.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39519.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41355.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39402.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37449.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41472.log.gz/var/log/ceph/ceph-client.admin.41355.log: 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41863.log 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39831.log.gz 2024-03-25T18:26:10.048 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.37449.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41355.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38774.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.37449.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41863.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34898.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38774.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39010.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.34898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41511.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40378.log: gzip -5 86.9% --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39010.log.gz 2024-03-25T18:26:10.049 INFO:teuthology.orchestra.run.smithi132.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.34898.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39127.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41511.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41785.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39127.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:gzip/var/log/ceph/ceph-client.admin.41785.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41004.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 --verbose -- 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41785.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: /var/log/ceph/ceph-client.admin.38893.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41316.log 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38893.log: gzip -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41004.log.gz 2024-03-25T18:26:10.050 INFO:teuthology.orchestra.run.smithi132.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41941.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41316.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38893.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40965.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41316.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41629.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40965.log: gzip -5 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41941.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41629.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40965.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.0.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41629.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41746.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35201.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41746.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41746.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.35201.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35201.log.gz 2024-03-25T18:26:10.051 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39049.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41824.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39049.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39049.log.gz 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39597.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40575.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39948.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40575.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39597.log.gz 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40575.log.gz 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39558.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39948.log.gz 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2024-03-25T18:26:10.052 INFO:teuthology.orchestra.run.smithi132.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40848.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39558.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5/var/log/ceph/ceph-client.admin.41394.log: --verbose -- /var/log/ceph/ceph-client.admin.38735.log 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40614.log 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.38735.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39205.log 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38735.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41433.log 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.39205.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40614.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39088.log 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr:gzip 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39205.log.gz 2024-03-25T18:26:10.053 INFO:teuthology.orchestra.run.smithi132.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40456.log 2024-03-25T18:26:10.054 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.41433.log: /var/log/ceph/ceph-client.admin.39088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35228.log 2024-03-25T18:26:10.054 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.40456.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39088.log.gz 2024-03-25T18:26:10.054 INFO:teuthology.orchestra.run.smithi132.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41433.log.gz 2024-03-25T18:26:10.054 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40456.log.gz 2024-03-25T18:26:10.054 INFO:teuthology.orchestra.run.smithi132.stderr: -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:26:10.054 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/ceph-client.admin.35228.log: /var/log/ceph/ceph-mon.c.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35228.log.gz 2024-03-25T18:26:10.100 INFO:teuthology.orchestra.run.smithi161.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:26:10.809 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:26:10.874 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:26:10.924 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:26:10.949 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:26:11.124 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:26:11.210 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:26:11.214 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:26:11.250 INFO:teuthology.orchestra.run.smithi132.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:26:14.808 INFO:teuthology.orchestra.run.smithi132.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:26:15.317 INFO:teuthology.orchestra.run.smithi161.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:26:15.318 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-25T18:26:15.318 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m5.312s 2024-03-25T18:26:15.319 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m8.933s 2024-03-25T18:26:15.319 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.334s 2024-03-25T18:26:15.413 INFO:teuthology.orchestra.run.smithi132.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:26:15.416 INFO:teuthology.orchestra.run.smithi132.stderr: 2024-03-25T18:26:15.416 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m5.442s 2024-03-25T18:26:15.416 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m14.158s 2024-03-25T18:26:15.416 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m0.538s 2024-03-25T18:26:15.416 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:26:15.416 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/remote/smithi132/log 2024-03-25T18:26:15.417 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:26:17.950 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/remote/smithi161/log 2024-03-25T18:26:17.951 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:26:19.615 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:19.837 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=712a3659dcc14f9e8ee147ecb38f2aaf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:19.842 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:26:19.855 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:26:19.863 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:26:19.864 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:26:19.879 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:26:19.962 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:26:19.971 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:26:19.971 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:26:19.974 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:26:19.987 INFO:teuthology.orchestra.run.smithi132.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:26:19.990 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:26:19.992 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:26:19.992 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2024-03-25T18:26:19.992 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +799us[ +799us] +/- 100ms 2024-03-25T18:26:19.992 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -1106us[-1109us] +/- 50ms 2024-03-25T18:26:19.993 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +908us[ +908us] +/- 70ms 2024-03-25T18:26:19.993 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:26:19.995 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:26:19.996 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-25T18:26:19.996 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 52 +783us[ +783us] +/- 100ms 2024-03-25T18:26:19.996 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -1103us[-1102us] +/- 50ms 2024-03-25T18:26:19.996 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +911us[ +912us] +/- 70ms 2024-03-25T18:26:19.996 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:26:19.997 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:26:20.006 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:26:20.006 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:26:20.014 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:26:20.104 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:26:20.178 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:26:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:20.209 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:26:20.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:20.241 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:26:20.251 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:26:20.264 INFO:teuthology.task.internal:Duration was 7691.215149 seconds 2024-03-25T18:26:20.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:26:20.274 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:26:20.275 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:26:20.278 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:26:20.319 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:26:20.327 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:26:20.665 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:26:20.665 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2024-03-25T18:26:20.666 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:26:20.725 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-03-25T18:26:20.726 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:26:20.762 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:26:20.763 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:26:20.768 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:26:20.832 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:26:20.833 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:26:20.837 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:26:20.978 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:26:20.989 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:26:20.989 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:26:21.016 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:26:21.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:26:21.055 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:21.058 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:21.081 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2024-03-25T18:26:21.109 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-03-25T18:26:21.133 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:21.160 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:26:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:26:21.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:26:21.197 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:26:21.197 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/remote/smithi132 2024-03-25T18:26:21.198 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:26:21.235 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260/remote/smithi161 2024-03-25T18:26:21.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:26:21.270 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:26:21.271 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:26:21.273 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:26:21.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:26:21.332 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:26:21.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:26:21.342 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:26:21.342 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:26:21.345 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:26:21.357 INFO:teuthology.orchestra.run.smithi132.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:26 /home/ubuntu/cephtest 2024-03-25T18:26:21.374 INFO:teuthology.orchestra.run.smithi161.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:26 /home/ubuntu/cephtest 2024-03-25T18:26:21.375 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:26:21.385 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:26:21.407 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:26:21.435 DEBUG:teuthology.nuke:shortname: smithi132 2024-03-25T18:26:21.436 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:26:21.461 DEBUG:teuthology.nuke:shortname: smithi161 2024-03-25T18:26:21.461 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:26:21.465 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:38.570206', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDyGPtVmN/t/hScr/Lh41lhd2ESJ/noBK6bWwPWjF49cnIjlANMDM1i0OWQJ541/5QR58IysledB17AshalkgKs='} 2024-03-25T18:26:21.490 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621260', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:38.568797', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC5lvloWX4M/z9MMgX4Co9KLPl1cnAOYTbRK0YDa1V8wVIVlmbo+CfY4RPrgGwOTj9EaNdDBbQRWlsZTtY4Fd28='} 2024-03-25T18:26:21.522 INFO:teuthology.orchestra.console.smithi132:Power off 2024-03-25T18:26:21.523 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:26:21.547 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2024-03-25T18:26:21.548 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:21.567 INFO:teuthology.orchestra.console.smithi161:Power off 2024-03-25T18:26:21.568 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:26:21.578 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-25T18:26:21.590 DEBUG:teuthology.orchestra.console.smithi161:power off output: Chassis Power Control: Down/Off 2024-03-25T18:26:21.590 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:21.618 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-25T18:26:25.579 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:25.619 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:25.691 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-25T18:26:25.730 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-25T18:26:29.694 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:29.732 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:29.808 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2024-03-25T18:26:29.842 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-25T18:26:33.810 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:33.843 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:26:33.922 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2024-03-25T18:26:33.953 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is off 2024-03-25T18:26:34.024 INFO:teuthology.orchestra.console.smithi132:Power off completed 2024-03-25T18:26:34.055 INFO:teuthology.orchestra.console.smithi161:Power off completed 2024-03-25T18:26:34.159 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7691.215148687363 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=712a3659dcc14f9e8ee147ecb38f2aaf status: fail success: false 2024-03-25T18:26:34.160 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:26:34.254 INFO:teuthology.run:FAIL